Jul 28 17:09:21.005179 ip-10-0-138-17 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:09:21.383532 ip-10-0-138-17 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:21.383532 ip-10-0-138-17 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:09:21.383532 ip-10-0-138-17 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:21.383532 ip-10-0-138-17 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:09:21.384178 ip-10-0-138-17 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:21.384717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.384580    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:09:21.390383 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390358    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.390383 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390382    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.390383 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390387    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.390383 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390391    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390395    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390399    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390402    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390405    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390408    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390411    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390414    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390417    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390420    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390423    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390425    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390428    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390431    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390435    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390438    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390441    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390444    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390446    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390449    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.390565 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390452    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390455    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390460    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390464    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390467    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390469    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390472    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390475    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390478    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390481    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390483    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390486    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390489    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390492    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390495    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390499    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390502    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390504    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390507    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.391099 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390510    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390513    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390518    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390523    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390527    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390530    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390533    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390536    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390540    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390543    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390546    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390549    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390552    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390555    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390558    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390562    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390565    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390569    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390572    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.391606 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390575    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390578    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390581    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390584    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390587    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390590    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390593    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390596    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390599    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390603    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390606    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390609    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390612    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390615    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390618    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390620    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390624    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390626    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390629    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390632    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.392185 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390635    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390637    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390641    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390644    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390647    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390649    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390652    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390656    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390659    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390662    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390665    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390667    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390670    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390673    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390676    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390679    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390681    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.390684    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392234    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392244    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.392706 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392248    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392252    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392255    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392258    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392262    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392265    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392268    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392271    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392273    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392277    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392279    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392283    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392286    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392288    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392292    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392295    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392298    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392301    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392304    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392307    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.393246 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392310    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392313    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392316    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392319    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392322    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392325    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392328    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392331    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392334    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392336    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392339    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392342    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392344    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392347    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392350    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392353    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392356    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392359    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392362    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.393781 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392365    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392367    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392371    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392374    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392378    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392382    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392385    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392390    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392393    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392414    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392419    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392423    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392428    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392431    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392434    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392437    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392440    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392443    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392445    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.394301 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392448    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392451    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392454    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392457    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392460    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392463    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392465    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392468    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392471    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392474    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392477    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392479    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392484    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392487    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392490    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392493    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392495    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392498    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392501    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392505    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.394795 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392507    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392511    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392515    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392519    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392522    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392525    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392528    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392531    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392534    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392537    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392540    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392543    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392545    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392550    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392554    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392557    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392560    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392563    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392566    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.392994    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.395343 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393000    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393003    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393007    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393010    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393013    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393016    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393018    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393021    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393024    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393027    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393030    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393033    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393036    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393040    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393043    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393047    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393049    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393053    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393056    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.395862 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393059    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393064    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393068    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393071    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393074    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393077    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393081    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393084    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393087    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393089    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393092    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393095    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393098    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393101    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393105    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393107    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393110    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393113    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393116    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.396394 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393118    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393121    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393124    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393126    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393129    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393132    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393135    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393138    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393140    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393143    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393147    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393150    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393155    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393158    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393161    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393164    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393166    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393170    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393172    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393175    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.396910 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393178    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393180    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393183    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393186    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393189    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393192    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393194    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393197    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393200    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393205    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393209    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393212    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393215    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393218    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393220    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393223    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393226    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393228    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393231    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.397491 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393234    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393237    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393239    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393242    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393246    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393251    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393254    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393257    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393260    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393263    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393266    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393269    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393271    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393274    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393277    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393280    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393283    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393285    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393288    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393291    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.398009 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.393294    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393370    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393379    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393386    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393391    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393395    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393399    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393403    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393408    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393412    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393415    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393419    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393423    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393426    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393429    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393434    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393437    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393442    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393446    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393451    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393454    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393457    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393461    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393465    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:09:21.398522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393469    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393473    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393476    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393480    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393483    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393487    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393490    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393493    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393496    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393501    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393505    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393508    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393511    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393514    2559 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393517    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393522    2559 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393525    2559 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393528    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393531    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393534    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393539    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393542    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393546    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393551    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393554    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:09:21.399163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393558    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393562    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393565    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393568    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393571    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393575    2559 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393579    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393582    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393586    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393589    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393592    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393596    2559 flags.go:64] FLAG: --help="false"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393599    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393602    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393605    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393608    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393612    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393616    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393619    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393622    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393625    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393629    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393632    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393635    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:09:21.399830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393638    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393642    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393644    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393648    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393651    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393654    2559 flags.go:64] FLAG: --lock-file=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393658    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393661    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393665    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393671    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393674    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393677    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393680    2559 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393683    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393687    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393690    2559 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393693    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393698    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393701    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393706    2559 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393709    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393712    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393715    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393718    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393721    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:09:21.400670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393725    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393728    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393736    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393739    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393742    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393745    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393748    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393754    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393757    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393761    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393764    2559 flags.go:64] FLAG: --port="10250"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393767    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393770    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b9fe4023c4afd4a7"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393774    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393779    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393783    2559 flags.go:64] FLAG: --register-node="true"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393786    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393790    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393794    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393797    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393800    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393809    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393813    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393816    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393819    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:09:21.401902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393822    2559 flags.go:64] FLAG: --runonce="false"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393825    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393829    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393832    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393835    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393838    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393842    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393845    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393848    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393851    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393854    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393858    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393861    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393864    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393868    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393871    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393876    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393879    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393882    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393887    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393890    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393894    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393899    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393902    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393906    2559 flags.go:64] FLAG: --v="2"
Jul 28 17:09:21.403065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393910    2559 flags.go:64] FLAG: --version="false"
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393915    2559 flags.go:64] FLAG: --vmodule=""
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393919    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.393923    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394085    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394090    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394093    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394096    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394099    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394101    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394104    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394107    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394110    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394113    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394115    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394118    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394121    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394124    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394126    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394129    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.403869 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394133    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394137    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394140    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394143    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394146    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394149    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394152    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394155    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394161    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394166    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394169    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394172    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394175    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394178    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394191    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394195    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394198    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394201    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394203    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394207    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.404808 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394210    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394213    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394215    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394218    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394221    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394224    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394227    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394229    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394232    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394235    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394238    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394242    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394244    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394247    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394250    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394252    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394255    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394258    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394261    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394264    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.405541 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394268    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394273    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394276    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394278    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394281    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394284    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394288    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394293    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394296    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394300    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394304    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394308    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394311    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394314    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394318    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394321    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394324    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394327    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394330    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.406447 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394333    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394336    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394338    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394341    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394344    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394347    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394350    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394352    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394355    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394358    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394361    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394363    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394366    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394370    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394374    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394377    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394380    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394383    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394386    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394389    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.407379 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394392    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394395    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394398    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.394401    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.394980    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.402781    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.402804    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.403575    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.403590    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403690    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403700    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403706    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403712    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403717    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.407985 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403725    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403732    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403738    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403743    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403748    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403753    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403758    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403763    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403767    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403772    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403777    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403782    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403786    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403791    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403797    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403801    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403806    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403810    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403815    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403821    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.408441 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403826    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403831    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403835    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403840    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403847    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403852    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403857    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403861    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403866    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403871    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403875    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403879    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403884    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403888    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403909    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403914    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403919    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403924    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403928    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403933    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.409123 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403937    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403941    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403946    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403950    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403954    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403959    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403964    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403969    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403973    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403978    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403982    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403987    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403992    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.403997    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404001    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404007    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404012    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404016    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404021    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404026    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.409719 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404031    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404036    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404043    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404049    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404054    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404059    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404066    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404073    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404078    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404083    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404087    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404093    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404097    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404102    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404106    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404111    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404116    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404120    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.410315 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404125    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404129    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404134    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404139    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404143    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404147    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404152    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404158    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404163    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404168    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404172    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404177    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404181    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404186    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404190    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.410840 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404195    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.404205    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404399    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404408    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404416    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404421    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404425    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404430    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404434    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404439    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404444    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404448    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404453    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404458    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:21.411274 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404463    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404468    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404472    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404477    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404481    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404486    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404490    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404494    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404499    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404504    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404508    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404514    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404519    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404523    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404528    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404533    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404537    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404541    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404546    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404550    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:21.411652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404555    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404560    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404564    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404569    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404573    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404578    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404582    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404587    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404591    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404596    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404600    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404605    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404609    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404614    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404618    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404622    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404627    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404632    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404637    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404641    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:21.412502 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404646    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404650    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404655    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404660    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404665    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404670    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404674    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404679    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404683    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404688    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404693    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404697    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404702    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404706    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404711    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404715    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404720    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404725    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404729    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404734    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:21.413323 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404739    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404744    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404748    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404752    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404757    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404762    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404769    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404776    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404781    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404786    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404790    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404794    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404799    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404804    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404808    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404812    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404819    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404826    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404832    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:21.413890 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404837    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404842    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404847    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404852    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404857    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404863    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404868    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:21.404873    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.404883    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.405062    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.407907    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.409597    2559 server.go:1019] "Starting client certificate rotation"
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.409712    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:21.414413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.409752    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:21.414786 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.412465    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:21.416726 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.416709    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:21.427123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.427101    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:09:21.429864 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.429841    2559 log.go:25] "Validated CRI v1 image API"
Jul 28 17:09:21.431268 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.431249    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:09:21.433804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.433783    2559 fs.go:135] Filesystem UUIDs: map[1ef82e82-cf0a-4c94-b744-bfa07df9b12a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e1a4ab9f-f505-490c-8623-2d1a3a3d9b38:/dev/nvme0n1p4]
Jul 28 17:09:21.433884 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.433803    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:09:21.439020 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.438999    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:21.439999 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.439842    2559 manager.go:216] Machine: {Timestamp:2026-07-28 17:09:21.437583514 +0000 UTC m=+0.319990635 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100179 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280690423a8291590f2186c1a28ea7 SystemUUID:ec280690-423a-8291-590f-2186c1a28ea7 BootID:cdb06d73-0e3b-467b-be02-419058819bc9 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:1f:6a:7c:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:1f:6a:7c:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:72:ae:b7:3f:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:09:21.439999 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.439984    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:09:21.440160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.440061    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:09:21.443408 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.443380    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:09:21.443551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.443411    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:09:21.444173 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.444155    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:09:21.444207 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.444176    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:09:21.444207 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.444185    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:21.444888 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.444877    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:21.445662 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.445651    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:09:21.447559 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447547    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:21.447672 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447661    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:09:21.447705 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447693    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:09:21.447705 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447703    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:09:21.447768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447719    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:09:21.447768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447728    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:09:21.447768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.447737    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:09:21.448102 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.448090    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:21.448153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.448109    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:21.450479 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.450463    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:09:21.452270 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.452256    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:09:21.452314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.452289    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:09:21.453686 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453673    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453691    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453697    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453703    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453714    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453720    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453728    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453734    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453740    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453746    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:09:21.453757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453760    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:09:21.454046 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.453766    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:09:21.455076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.455060    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:09:21.455119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.455080    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:09:21.458495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.458475    2559 server.go:1268] "Started kubelet"
Jul 28 17:09:21.458615 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.458568    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:09:21.459384 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.459327    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:09:21.459469 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.459420    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:09:21.459480 ip-10-0-138-17 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:09:21.460349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.460081    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:09:21.460569 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.460540    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:09:21.460627 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.460585    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:09:21.461027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.461012    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:09:21.462886 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.462865    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:09:21.464688 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.464665    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:09:21.469039 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.467981    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-17.ec2.internal.18c68333f3f45f2c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-17.ec2.internal,UID:ip-10-0-138-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-17.ec2.internal,},FirstTimestamp:2026-07-28 17:09:21.45842974 +0000 UTC m=+0.340836851,LastTimestamp:2026-07-28 17:09:21.45842974 +0000 UTC m=+0.340836851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-17.ec2.internal,}"
Jul 28 17:09:21.469238 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469218    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:09:21.469346 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469220    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:21.469346 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.469302    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.469822 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469803    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:09:21.469822 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469824    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:09:21.469976 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469945    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:09:21.469976 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469973    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:09:21.470075 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.469982    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:09:21.470075 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470044    2559 factory.go:55] Registering systemd factory
Jul 28 17:09:21.470075 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470060    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:09:21.470868 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470834    2559 factory.go:153] Registering CRI-O factory
Jul 28 17:09:21.470945 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470872    2559 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:09:21.470945 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470947    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:09:21.471094 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470978    2559 factory.go:103] Registering Raw factory
Jul 28 17:09:21.471094 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.470997    2559 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:09:21.472631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.472612    2559 manager.go:349] Starting recovery of all containers
Jul 28 17:09:21.478642 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.478617    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:09:21.478780 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.478749    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:09:21.484743 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.484717    2559 manager.go:354] Recovery completed
Jul 28 17:09:21.489300 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.489283    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.491142 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491121    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.491225 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491155    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.491225 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491166    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.491721 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491705    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:09:21.491721 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491719    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:09:21.491814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491736    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:21.491814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.491720    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.492038 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.492020    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6bh6t"
Jul 28 17:09:21.492569 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.492553    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.492644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.492585    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.492644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.492600    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.493496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.493479    2559 policy_none.go:49] "None policy: Start"
Jul 28 17:09:21.493566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.493508    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:09:21.493566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.493520    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:09:21.493666 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.493574    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-17.ec2.internal.18c68333f5e781ad  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-17.ec2.internal,UID:ip-10-0-138-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-17.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-17.ec2.internal,},FirstTimestamp:2026-07-28 17:09:21.491141037 +0000 UTC m=+0.373548149,LastTimestamp:2026-07-28 17:09:21.491141037 +0000 UTC m=+0.373548149,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-17.ec2.internal,}"
Jul 28 17:09:21.494541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.494529    2559 policy_none.go:47] "Start"
Jul 28 17:09:21.501086 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.500974    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6bh6t"
Jul 28 17:09:21.502660 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.502590    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-17.ec2.internal.18c68333f5e7cf0b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-17.ec2.internal,UID:ip-10-0-138-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-17.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-17.ec2.internal,},FirstTimestamp:2026-07-28 17:09:21.491160843 +0000 UTC m=+0.373567953,LastTimestamp:2026-07-28 17:09:21.491160843 +0000 UTC m=+0.373567953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-17.ec2.internal,}"
Jul 28 17:09:21.505129 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.505063    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-17.ec2.internal.18c68333f5e7f476  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-17.ec2.internal,UID:ip-10-0-138-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-17.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-17.ec2.internal,},FirstTimestamp:2026-07-28 17:09:21.491170422 +0000 UTC m=+0.373577533,LastTimestamp:2026-07-28 17:09:21.491170422 +0000 UTC m=+0.373577533,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-17.ec2.internal,}"
Jul 28 17:09:21.527525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.527509    2559 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:09:21.527605 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.527540    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:09:21.527605 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.527552    2559 server.go:87] "Starting device plugin registration server"
Jul 28 17:09:21.527818 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.527806    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:09:21.527853 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.527822    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:09:21.527984 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.527967    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:09:21.528067 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.528052    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:09:21.528140 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.528068    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:09:21.528538 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.528512    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:09:21.528615 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.528570    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.628273 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.628245    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.629354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.629144    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.629354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.629178    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.629354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.629189    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.629354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.629217    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.639233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.639212    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.639285 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.639239    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-17.ec2.internal\": node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.665093 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.665061    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:09:21.665445 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.665425    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.666339 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.666323    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:09:21.666396 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.666361    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:09:21.666396 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.666380    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:09:21.666475 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.666415    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:09:21.668584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.668560    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:21.766011 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.765973    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.767159 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.767141    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"]
Jul 28 17:09:21.770437 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.767199    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.771843 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.771824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.771909 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.771853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.774348 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.774334    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.774398 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.774362    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.774398 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.774379    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.775749 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.775737    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.775856 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.775839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.776356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.776090    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.776839    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.776875    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.776891    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.777261    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.777294    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.778314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.777309    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.780881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.780852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.780982 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.780886    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:21.781756 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.781740    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:21.781844 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.781763    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:21.781844 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.781778    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:21.811729 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.811707    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-17.ec2.internal\" not found" node="ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.816498 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.816481    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-17.ec2.internal\" not found" node="ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.866995 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.866974    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.872793 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.872775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.872855 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.872804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.872855 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.872829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1a690d26e46a9b8ac8c0a59f590243d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-17.ec2.internal\" (UID: \"a1a690d26e46a9b8ac8c0a59f590243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.872855 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.872839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.872986 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.872896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec44b9d7dbc7dbb1f61524a3eec673-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal\" (UID: \"9fec44b9d7dbc7dbb1f61524a3eec673\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.967422 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:21.967339    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:21.973088 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.973058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1a690d26e46a9b8ac8c0a59f590243d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-17.ec2.internal\" (UID: \"a1a690d26e46a9b8ac8c0a59f590243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:21.973152 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:21.973123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1a690d26e46a9b8ac8c0a59f590243d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-17.ec2.internal\" (UID: \"a1a690d26e46a9b8ac8c0a59f590243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:22.074767 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.074719    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:22.116029 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.115999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:22.119698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.119680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:22.175621 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.175593    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:22.276264 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.276203    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:22.377196 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.377159    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-17.ec2.internal\" not found"
Jul 28 17:09:22.410073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.410046    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:09:22.410491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.410218    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:09:22.430954 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.430932    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:22.448790 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.448769    2559 apiserver.go:52] "Watching apiserver"
Jul 28 17:09:22.459524 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.459507    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:09:22.460780 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.460759    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-pmvhk","kube-system/global-pull-secret-syncer-bztwt","kube-system/konnectivity-agent-pzwwh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x","openshift-cluster-node-tuning-operator/tuned-7n85v","openshift-dns/node-resolver-vvb2g","openshift-multus/multus-additional-cni-plugins-w8gdg","openshift-multus/multus-fhz2t","openshift-network-operator/iptables-alerter-k2qcw","openshift-image-registry/node-ca-bfbzv","openshift-multus/network-metrics-daemon-zk8h7","openshift-network-diagnostics/network-check-target-jtqn7"]
Jul 28 17:09:22.462479 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.462460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.463688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.463665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.463768 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.463740    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:22.464874 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.464854    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.464981 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.464893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.465256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465086    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.465256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:09:22.465422 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:09:22.465484 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465454    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:09:22.465484 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465467    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:09:22.465586 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.465483    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-njvj2\""
Jul 28 17:09:22.466206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.466192    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.467169 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.467149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-glp7v\""
Jul 28 17:09:22.467241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.467226    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:09:22.467463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.467447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:09:22.467524 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.467484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.468377 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.468364    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.468755 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.468742    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.468964 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.468947    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:09:22.468964 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.468960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.469040 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.468966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pt99j\""
Jul 28 17:09:22.469336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.469321    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:22.469620 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.469608    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:22.469893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.469873    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.470006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.469990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.470073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.470043    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g64rv\""
Jul 28 17:09:22.470416 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.470400    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.471623 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.471597    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lzdzb\""
Jul 28 17:09:22.472198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.472085    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.472198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.472129    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.473532 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.472794    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.473532 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.473237    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:09:22.473532 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.473522    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:09:22.475054 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.473836    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.475054 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.473880    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pzmm8\""
Jul 28 17:09:22.475054 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.474211    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:09:22.475299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.475139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.475371 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.475350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.476290 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.476271    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4q7f4\""
Jul 28 17:09:22.476392 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.476274    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:09:22.477291 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-system-cni-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.477354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-agent-certs\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.477354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-device-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.477458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-lib-modules\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.477458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-config\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.477458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-script-lib\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.477458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-run\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-dbus\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-sys\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-hosts-file\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-tmp-dir\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.477614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-kubelet-config\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-os-release\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-sys-fs\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbff\" (UniqueName: \"kubernetes.io/projected/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-kube-api-access-thbff\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-systemd-units\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovn-node-metrics-cert\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysconfig\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.477890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-bin\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-netd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.477980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-socket-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-conf\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhmc\" (UniqueName: \"kubernetes.io/projected/26e75bc2-777e-4c20-80fe-5ee167820ece-kube-api-access-bbhmc\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-etc-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-kubernetes\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ws7\" (UniqueName: \"kubernetes.io/projected/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-kube-api-access-b6ws7\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5xf\" (UniqueName: \"kubernetes.io/projected/da4dbe31-ef2d-4a5f-a645-29079110aa50-kube-api-access-mz5xf\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-konnectivity-ca\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-registration-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-kubelet\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-netns\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478562    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-ovn\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.478631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-log-socket\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cnibin\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478749    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478766    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jch8b\""
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-systemd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-env-overrides\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-modprobe-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-var-lib-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.478991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-tuned\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-tmp\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-node-log\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479053    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-slash\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.479546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgc26\" (UniqueName: \"kubernetes.io/projected/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kube-api-access-zgc26\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-var-lib-kubelet\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-systemd\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-host\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.479483    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.479551    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.480047    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.480202    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.480243    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7n8hl\""
Jul 28 17:09:22.480885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.480354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:09:22.481282 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.481264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:22.481340 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.481321    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:22.493968 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.493947    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:22.494040 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.494029    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"
Jul 28 17:09:22.494133 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.494117    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal"]
Jul 28 17:09:22.497543 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.497527    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:22.504528 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.504484    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:04:22 +0000 UTC" deadline="2028-03-07 21:54:48.444125591 +0000 UTC"
Jul 28 17:09:22.504528 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.504524    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14116h45m25.939603897s"
Jul 28 17:09:22.522066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.522040    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:22.522350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.522336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal"]
Jul 28 17:09:22.527394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.527355    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8szd"
Jul 28 17:09:22.536933 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.536916    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8szd"
Jul 28 17:09:22.571464 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.571440    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:09:22.579960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.579931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-hosts-file\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.580052 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.579968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-netns\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.580052 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.579994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hl9\" (UniqueName: \"kubernetes.io/projected/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-kube-api-access-r4hl9\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.580052 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-hosts-file\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.580202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5af3a59-9f62-4254-b97e-ba7486e1cf54-iptables-alerter-script\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.580202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-host\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.580202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-tmp-dir\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.580202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-kubelet-config\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-os-release\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-kubelet-config\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580291    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-os-release\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-sys-fs\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbff\" (UniqueName: \"kubernetes.io/projected/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-kube-api-access-thbff\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-systemd-units\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovn-node-metrics-cert\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-tmp-dir\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysconfig\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.580471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-systemd-units\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysconfig\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-sys-fs\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.580621    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-k8s-cni-cncf-io\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.580705    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.080660003 +0000 UTC m=+1.963067099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm6j\" (UniqueName: \"kubernetes.io/projected/5207cf1f-96b1-42b8-8bd1-623cfc669465-kube-api-access-7jm6j\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-bin\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-netd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580839    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-socket-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-conf\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhmc\" (UniqueName: \"kubernetes.io/projected/26e75bc2-777e-4c20-80fe-5ee167820ece-kube-api-access-bbhmc\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-netd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-bin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.580852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-cni-bin\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5af3a59-9f62-4254-b97e-ba7486e1cf54-host-slash\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-etc-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-conf\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-kubernetes\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.581840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ws7\" (UniqueName: \"kubernetes.io/projected/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-kube-api-access-b6ws7\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.581986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5xf\" (UniqueName: \"kubernetes.io/projected/da4dbe31-ef2d-4a5f-a645-29079110aa50-kube-api-access-mz5xf\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-socket-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-etc-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-konnectivity-ca\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-kubernetes\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-registration-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-kubelet\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-registration-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.582511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-kubelet\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-netns\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-ovn\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-log-socket\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cnibin\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582819    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-konnectivity-ca\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.583069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-kubelet\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-ovn\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.582949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-netns\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.583000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-log-socket\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.583136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-run-ovn-kubernetes\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.583271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cnibin\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.585349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.583292    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.583777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585296    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-etc-kubernetes\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tzp\" (UniqueName: \"kubernetes.io/projected/f5af3a59-9f62-4254-b97e-ba7486e1cf54-kube-api-access-74tzp\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-systemd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-env-overrides\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.585728 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-systemd\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.585677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.586137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-modprobe-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.586137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.586137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-multus\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.586137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovn-node-metrics-cert\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-var-lib-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-socket-dir-parent\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-hostroot\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-var-lib-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-modprobe-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-serviceca\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-tuned\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-tmp\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-node-log\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-slash\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.586387 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-env-overrides\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgc26\" (UniqueName: \"kubernetes.io/projected/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kube-api-access-zgc26\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-var-lib-kubelet\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-host-slash\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-var-lib-kubelet\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-run-openvswitch\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da4dbe31-ef2d-4a5f-a645-29079110aa50-node-log\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrj7n\" (UniqueName: \"kubernetes.io/projected/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-kube-api-access-vrj7n\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-os-release\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-systemd\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-host\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-system-cni-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-host\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-systemd\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-conf-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:22.587027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-system-cni-dir\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-agent-certs\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-device-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-lib-modules\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-system-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3eff3acd-73cb-48ec-bf7c-332fe083b72d-device-dir\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.586951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-daemon-config\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-cnibin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-lib-modules\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-cni-binary-copy\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-config\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-script-lib\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-run\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-run\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-dbus\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-multus-certs\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-sys\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.587994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-dbus\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:22.588910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-sys\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.588910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-sysctl-d\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.588910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.587728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-script-lib\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.588910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.588698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da4dbe31-ef2d-4a5f-a645-29079110aa50-ovnkube-config\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.588910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.588814    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-etc-tuned\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.589199 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.588920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26e75bc2-777e-4c20-80fe-5ee167820ece-tmp\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.589526 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.589496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99abb92a-6ff6-4307-af4f-7f8b0fa3674b-agent-certs\") pod \"konnectivity-agent-pzwwh\" (UID: \"99abb92a-6ff6-4307-af4f-7f8b0fa3674b\") " pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.589704 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.589685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbff\" (UniqueName: \"kubernetes.io/projected/dab1330b-69d5-4f1c-9266-1ae217f6b1c7-kube-api-access-thbff\") pod \"multus-additional-cni-plugins-w8gdg\" (UID: \"dab1330b-69d5-4f1c-9266-1ae217f6b1c7\") " pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.591296 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.591268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ws7\" (UniqueName: \"kubernetes.io/projected/15a784d6-4b9f-48f7-b7af-95afd4bcfce4-kube-api-access-b6ws7\") pod \"node-resolver-vvb2g\" (UID: \"15a784d6-4b9f-48f7-b7af-95afd4bcfce4\") " pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.592283 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.592267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5xf\" (UniqueName: \"kubernetes.io/projected/da4dbe31-ef2d-4a5f-a645-29079110aa50-kube-api-access-mz5xf\") pod \"ovnkube-node-pmvhk\" (UID: \"da4dbe31-ef2d-4a5f-a645-29079110aa50\") " pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.592491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.592458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhmc\" (UniqueName: \"kubernetes.io/projected/26e75bc2-777e-4c20-80fe-5ee167820ece-kube-api-access-bbhmc\") pod \"tuned-7n85v\" (UID: \"26e75bc2-777e-4c20-80fe-5ee167820ece\") " pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.593961 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.593930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgc26\" (UniqueName: \"kubernetes.io/projected/3eff3acd-73cb-48ec-bf7c-332fe083b72d-kube-api-access-zgc26\") pod \"aws-ebs-csi-driver-node-6fj2x\" (UID: \"3eff3acd-73cb-48ec-bf7c-332fe083b72d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.687576 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-netns\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hl9\" (UniqueName: \"kubernetes.io/projected/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-kube-api-access-r4hl9\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.687698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687598    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5af3a59-9f62-4254-b97e-ba7486e1cf54-iptables-alerter-script\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.687698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-host\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.687698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687677    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-netns\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-host\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-k8s-cni-cncf-io\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm6j\" (UniqueName: \"kubernetes.io/projected/5207cf1f-96b1-42b8-8bd1-623cfc669465-kube-api-access-7jm6j\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-k8s-cni-cncf-io\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-bin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-bin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.687891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5af3a59-9f62-4254-b97e-ba7486e1cf54-host-slash\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.687911    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-kubelet\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-etc-kubernetes\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.687971    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.187940572 +0000 UTC m=+2.070347667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.687992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-etc-kubernetes\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5af3a59-9f62-4254-b97e-ba7486e1cf54-host-slash\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-kubelet\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tzp\" (UniqueName: \"kubernetes.io/projected/f5af3a59-9f62-4254-b97e-ba7486e1cf54-kube-api-access-74tzp\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-multus\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-socket-dir-parent\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-var-lib-cni-multus\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-hostroot\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-serviceca\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-socket-dir-parent\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-hostroot\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5af3a59-9f62-4254-b97e-ba7486e1cf54-iptables-alerter-script\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj7n\" (UniqueName: \"kubernetes.io/projected/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-kube-api-access-vrj7n\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-os-release\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-conf-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-os-release\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-conf-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-system-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-daemon-config\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-cnibin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-system-cni-dir\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-cnibin\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-cni-binary-copy\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-multus-certs\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5207cf1f-96b1-42b8-8bd1-623cfc669465-host-run-multus-certs\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.688797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-serviceca\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.689284 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-multus-daemon-config\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.689284 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.688950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5207cf1f-96b1-42b8-8bd1-623cfc669465-cni-binary-copy\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.696968 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.696938    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:22.697048 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.696972    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:22.697048 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.696988    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.697048 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:22.697044    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.197028609 +0000 UTC m=+2.079435709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:22.699494 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.699463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm6j\" (UniqueName: \"kubernetes.io/projected/5207cf1f-96b1-42b8-8bd1-623cfc669465-kube-api-access-7jm6j\") pod \"multus-fhz2t\" (UID: \"5207cf1f-96b1-42b8-8bd1-623cfc669465\") " pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.699597 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.699509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tzp\" (UniqueName: \"kubernetes.io/projected/f5af3a59-9f62-4254-b97e-ba7486e1cf54-kube-api-access-74tzp\") pod \"iptables-alerter-k2qcw\" (UID: \"f5af3a59-9f62-4254-b97e-ba7486e1cf54\") " pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.699597 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.699513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj7n\" (UniqueName: \"kubernetes.io/projected/66cd0b5b-1bba-4a4b-9a37-79d8e241db87-kube-api-access-vrj7n\") pod \"node-ca-bfbzv\" (UID: \"66cd0b5b-1bba-4a4b-9a37-79d8e241db87\") " pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.700072 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.700053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hl9\" (UniqueName: \"kubernetes.io/projected/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-kube-api-access-r4hl9\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:22.792215 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.792142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:22.806240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.806209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:22.824399 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.824368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x"
Jul 28 17:09:22.841143 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.841126    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7n85v"
Jul 28 17:09:22.849407 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.849382    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eff3acd_73cb_48ec_bf7c_332fe083b72d.slice/crio-1c32b7fa64c48e600fcfd13da2a75da6784b5933d8bd5f31587a95b49e168517 WatchSource:0}: Error finding container 1c32b7fa64c48e600fcfd13da2a75da6784b5933d8bd5f31587a95b49e168517: Status 404 returned error can't find the container with id 1c32b7fa64c48e600fcfd13da2a75da6784b5933d8bd5f31587a95b49e168517
Jul 28 17:09:22.849742 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.849722    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a690d26e46a9b8ac8c0a59f590243d.slice/crio-efccdf9d3ca9b9eac32ca078d7e9dc9cb1f61dddc2c2bf672367dbfba500227b WatchSource:0}: Error finding container efccdf9d3ca9b9eac32ca078d7e9dc9cb1f61dddc2c2bf672367dbfba500227b: Status 404 returned error can't find the container with id efccdf9d3ca9b9eac32ca078d7e9dc9cb1f61dddc2c2bf672367dbfba500227b
Jul 28 17:09:22.850892 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.850875    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e75bc2_777e_4c20_80fe_5ee167820ece.slice/crio-5c593d1f05d1a05b956c4329609392838854ec72ec86cb06c58e8a9b0698b581 WatchSource:0}: Error finding container 5c593d1f05d1a05b956c4329609392838854ec72ec86cb06c58e8a9b0698b581: Status 404 returned error can't find the container with id 5c593d1f05d1a05b956c4329609392838854ec72ec86cb06c58e8a9b0698b581
Jul 28 17:09:22.855269 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.855250    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:09:22.859099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.859083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vvb2g"
Jul 28 17:09:22.863600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.863581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w8gdg"
Jul 28 17:09:22.867169 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.867151    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a784d6_4b9f_48f7_b7af_95afd4bcfce4.slice/crio-b970feae3ee049d6b0f0fe00465e3820ce52b0f0a0b4455b329113a435e349fd WatchSource:0}: Error finding container b970feae3ee049d6b0f0fe00465e3820ce52b0f0a0b4455b329113a435e349fd: Status 404 returned error can't find the container with id b970feae3ee049d6b0f0fe00465e3820ce52b0f0a0b4455b329113a435e349fd
Jul 28 17:09:22.871666 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.871641    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab1330b_69d5_4f1c_9266_1ae217f6b1c7.slice/crio-61df9fcc7c714b5e64f75aa37a7e9c63010d25f9a0368dfaf94096bb7d41ff1f WatchSource:0}: Error finding container 61df9fcc7c714b5e64f75aa37a7e9c63010d25f9a0368dfaf94096bb7d41ff1f: Status 404 returned error can't find the container with id 61df9fcc7c714b5e64f75aa37a7e9c63010d25f9a0368dfaf94096bb7d41ff1f
Jul 28 17:09:22.886776 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.886759    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fhz2t"
Jul 28 17:09:22.893839 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.893818    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5207cf1f_96b1_42b8_8bd1_623cfc669465.slice/crio-50a4aaf44d84c79b71ff14314eec9184440ac261155ddc89fe38e45e9265a390 WatchSource:0}: Error finding container 50a4aaf44d84c79b71ff14314eec9184440ac261155ddc89fe38e45e9265a390: Status 404 returned error can't find the container with id 50a4aaf44d84c79b71ff14314eec9184440ac261155ddc89fe38e45e9265a390
Jul 28 17:09:22.916876 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.916854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2qcw"
Jul 28 17:09:22.921896 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.921863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfbzv"
Jul 28 17:09:22.923609 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.923590    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5af3a59_9f62_4254_b97e_ba7486e1cf54.slice/crio-b4425ebc6d6f270bd788ee8957ef313d83a460121322dbf5534735ea6b1327df WatchSource:0}: Error finding container b4425ebc6d6f270bd788ee8957ef313d83a460121322dbf5534735ea6b1327df: Status 404 returned error can't find the container with id b4425ebc6d6f270bd788ee8957ef313d83a460121322dbf5534735ea6b1327df
Jul 28 17:09:22.929232 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:22.929213    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cd0b5b_1bba_4a4b_9a37_79d8e241db87.slice/crio-6a0bb62c543315bc171a1dbf56fbd703b68005b13c85bc9dd7e90821705a0520 WatchSource:0}: Error finding container 6a0bb62c543315bc171a1dbf56fbd703b68005b13c85bc9dd7e90821705a0520: Status 404 returned error can't find the container with id 6a0bb62c543315bc171a1dbf56fbd703b68005b13c85bc9dd7e90821705a0520
Jul 28 17:09:22.985253 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.985230    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:22.995482 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:22.995464    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:23.073793 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:23.073764    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fec44b9d7dbc7dbb1f61524a3eec673.slice/crio-646dc2bb36a0b3f881ed1db1570e7c8d8312765ddcb65d4004e97d3e21338c1c WatchSource:0}: Error finding container 646dc2bb36a0b3f881ed1db1570e7c8d8312765ddcb65d4004e97d3e21338c1c: Status 404 returned error can't find the container with id 646dc2bb36a0b3f881ed1db1570e7c8d8312765ddcb65d4004e97d3e21338c1c
Jul 28 17:09:23.092153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.092130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:23.092285 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.092267    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:23.092336 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.092328    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:24.092311976 +0000 UTC m=+2.974719076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:23.154534 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:23.154510    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4dbe31_ef2d_4a5f_a645_29079110aa50.slice/crio-dbbb52f471b621c843b2fbbaf8fdb3e37fd3329f60a96f8a28549434622087f4 WatchSource:0}: Error finding container dbbb52f471b621c843b2fbbaf8fdb3e37fd3329f60a96f8a28549434622087f4: Status 404 returned error can't find the container with id dbbb52f471b621c843b2fbbaf8fdb3e37fd3329f60a96f8a28549434622087f4
Jul 28 17:09:23.193147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.193119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:23.193262 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.193251    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:23.193310 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.193298    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:24.193283432 +0000 UTC m=+3.075690527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:23.293516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.293478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:23.293706 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.293633    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:23.293706 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.293652    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:23.293706 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.293665    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:23.293877 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.293721    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:24.293702379 +0000 UTC m=+3.176109490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:23.423732 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.423654    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:23.539447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.539387    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:23 +0000 UTC" deadline="2028-05-13 05:02:52.535722538 +0000 UTC"
Jul 28 17:09:23.539447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.539441    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15707h53m28.996284109s"
Jul 28 17:09:23.669624 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.669589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:23.669822 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.669755    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:23.670228 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.670206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:23.670334 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.670313    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:23.670397 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.670384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:23.670473 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:23.670455    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:23.687681 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.687579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"dbbb52f471b621c843b2fbbaf8fdb3e37fd3329f60a96f8a28549434622087f4"}
Jul 28 17:09:23.697081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.696650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal" event={"ID":"9fec44b9d7dbc7dbb1f61524a3eec673","Type":"ContainerStarted","Data":"646dc2bb36a0b3f881ed1db1570e7c8d8312765ddcb65d4004e97d3e21338c1c"}
Jul 28 17:09:23.702707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.702673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfbzv" event={"ID":"66cd0b5b-1bba-4a4b-9a37-79d8e241db87","Type":"ContainerStarted","Data":"6a0bb62c543315bc171a1dbf56fbd703b68005b13c85bc9dd7e90821705a0520"}
Jul 28 17:09:23.704592 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.704565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2qcw" event={"ID":"f5af3a59-9f62-4254-b97e-ba7486e1cf54","Type":"ContainerStarted","Data":"b4425ebc6d6f270bd788ee8957ef313d83a460121322dbf5534735ea6b1327df"}
Jul 28 17:09:23.712608 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.711840    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerStarted","Data":"61df9fcc7c714b5e64f75aa37a7e9c63010d25f9a0368dfaf94096bb7d41ff1f"}
Jul 28 17:09:23.718167 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.718122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vvb2g" event={"ID":"15a784d6-4b9f-48f7-b7af-95afd4bcfce4","Type":"ContainerStarted","Data":"b970feae3ee049d6b0f0fe00465e3820ce52b0f0a0b4455b329113a435e349fd"}
Jul 28 17:09:23.725588 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.725553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhz2t" event={"ID":"5207cf1f-96b1-42b8-8bd1-623cfc669465","Type":"ContainerStarted","Data":"50a4aaf44d84c79b71ff14314eec9184440ac261155ddc89fe38e45e9265a390"}
Jul 28 17:09:23.737032 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.737001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7n85v" event={"ID":"26e75bc2-777e-4c20-80fe-5ee167820ece","Type":"ContainerStarted","Data":"5c593d1f05d1a05b956c4329609392838854ec72ec86cb06c58e8a9b0698b581"}
Jul 28 17:09:23.752008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.751969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal" event={"ID":"a1a690d26e46a9b8ac8c0a59f590243d","Type":"ContainerStarted","Data":"efccdf9d3ca9b9eac32ca078d7e9dc9cb1f61dddc2c2bf672367dbfba500227b"}
Jul 28 17:09:23.758288 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:23.758246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x" event={"ID":"3eff3acd-73cb-48ec-bf7c-332fe083b72d","Type":"ContainerStarted","Data":"1c32b7fa64c48e600fcfd13da2a75da6784b5933d8bd5f31587a95b49e168517"}
Jul 28 17:09:24.100278 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.100186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:24.100466 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.100430    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:24.100546 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.100506    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.100478429 +0000 UTC m=+4.982885540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:24.201670 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.201630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:24.201870 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.201821    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:24.201968 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.201890    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.201869325 +0000 UTC m=+5.084276421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:24.302875 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.302835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:24.303125 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.303104    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:24.303212 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.303139    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:24.303212 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.303155    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:24.303324 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:24.303224    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:26.303203627 +0000 UTC m=+5.185610737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:24.472042 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:24.472005    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99abb92a_6ff6_4307_af4f_7f8b0fa3674b.slice/crio-99fa07397063adb22cddef7446b3bcd4397160a31af350871d4851b5361d7b74 WatchSource:0}: Error finding container 99fa07397063adb22cddef7446b3bcd4397160a31af350871d4851b5361d7b74: Status 404 returned error can't find the container with id 99fa07397063adb22cddef7446b3bcd4397160a31af350871d4851b5361d7b74
Jul 28 17:09:24.539980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.539934    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:23 +0000 UTC" deadline="2028-03-05 09:21:38.045596248 +0000 UTC"
Jul 28 17:09:24.539980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.539974    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14056h12m13.505627582s"
Jul 28 17:09:24.769210 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:24.769069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzwwh" event={"ID":"99abb92a-6ff6-4307-af4f-7f8b0fa3674b","Type":"ContainerStarted","Data":"99fa07397063adb22cddef7446b3bcd4397160a31af350871d4851b5361d7b74"}
Jul 28 17:09:25.666864 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:25.666770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:25.667352 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:25.666927    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:25.667352 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:25.667304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:25.667352 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:25.667330    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:25.667530 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:25.667435    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:25.667588 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:25.667536    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:26.120847 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:26.120806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:26.121070 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.121039    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:26.121140 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.121111    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:30.121089298 +0000 UTC m=+9.003496416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:26.221270 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:26.221227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:26.221465 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.221435    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:26.221538 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.221501    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:30.221481029 +0000 UTC m=+9.103888124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:26.322928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:26.322211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:26.322928 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.322445    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:26.322928 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.322471    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:26.322928 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.322484    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:26.322928 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:26.322551    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:30.322531198 +0000 UTC m=+9.204938295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:27.667694 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:27.667100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:27.667694 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:27.667262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:27.667694 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:27.667570    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:27.667694 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:27.667687    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:27.668471 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:27.667888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:27.668471 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:27.667975    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:29.671202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:29.671249    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:29.671411    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:29.671838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:29.672006    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:29.673530 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:29.672169    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:30.157359 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:30.157271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:30.157533 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.157461    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:30.157533 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.157524    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:38.157503006 +0000 UTC m=+17.039910106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:30.257987 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:30.257944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:30.258189 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.258144    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:30.258260 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.258225    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:38.258204688 +0000 UTC m=+17.140611788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:30.358544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:30.358510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:30.358785 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.358688    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:30.358785 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.358709    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:30.358785 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.358728    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:30.358785 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:30.358785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:38.358766941 +0000 UTC m=+17.241174042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:31.668592 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:31.668505    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:31.668592 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:31.668614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:31.669255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:31.668653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:31.669255 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:31.668747    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:31.669255 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:31.669126    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:31.669255 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:31.669230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:33.670669 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:33.670628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:33.671174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:33.670628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:33.671174 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:33.670754    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:33.671174 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:33.670852    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:33.671174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:33.670628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:33.671174 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:33.670965    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:35.667101 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:35.666914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:35.667537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:35.666922    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:35.667537 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:35.667217    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:35.667537 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:35.667278    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:35.667537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:35.666991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:35.667537 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:35.667367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:37.667653 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:37.667619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:37.668102 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:37.667619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:37.668102 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:37.667764    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:37.668102 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:37.667619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:37.668102 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:37.667821    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:37.668102 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:37.667911    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:38.223773 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:38.223737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:38.223989 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.223909    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:38.223989 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.223982    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.223962057 +0000 UTC m=+33.106369151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:38.324615 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:38.324584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:38.324771 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.324693    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:38.324771 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.324759    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.324741485 +0000 UTC m=+33.207148581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:38.425393 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:38.425357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:38.425564 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.425505    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:38.425564 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.425533    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:38.425564 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.425544    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:38.425735 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:38.425603    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.425585701 +0000 UTC m=+33.307992795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:39.669842 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:39.669811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:39.670297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:39.669939    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:39.670297 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:39.669950    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:39.670297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:39.669971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:39.670297 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:39.670059    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:39.670297 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:39.670135    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:41.667942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:41.667907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:41.668405 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:41.667955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:41.668405 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:41.668037    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:41.668505 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:41.668440    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:41.668662 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:41.668640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:41.668773 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:41.668731    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:43.667063 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.667037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:43.667434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.667048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:43.667434 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:43.667150    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:43.667434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.667048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:43.667434 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:43.667238    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:43.667434 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:43.667284    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:43.815790 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.815756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fhz2t" event={"ID":"5207cf1f-96b1-42b8-8bd1-623cfc669465","Type":"ContainerStarted","Data":"8aff1ddd30ab886dc69af5f3840896dd3d7ba2b6d317df7932de7b968c97b6e9"}
Jul 28 17:09:43.817236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.817211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7n85v" event={"ID":"26e75bc2-777e-4c20-80fe-5ee167820ece","Type":"ContainerStarted","Data":"a9c20663bde4ff7a6cb6566b872cce4046e52fb1fc9f10c9e7c82fa63eab438e"}
Jul 28 17:09:43.818629 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.818599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal" event={"ID":"a1a690d26e46a9b8ac8c0a59f590243d","Type":"ContainerStarted","Data":"d4289971b75fc57c3de8505b00199de788fe81ebbe8e270b46d1fcbc3b210dc7"}
Jul 28 17:09:43.845413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.845354    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fhz2t" podStartSLOduration=1.132824272 podStartE2EDuration="21.845339242s" totalImagesPullingTime="20.71251497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:43.832528598 +0000 UTC m=+22.714935754" watchObservedRunningTime="2026-07-28 17:09:43.845339242 +0000 UTC m=+22.727746336"
Jul 28 17:09:43.845812 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.845784    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-17.ec2.internal" podStartSLOduration=20.845776448 podStartE2EDuration="20.845776448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:43.845411356 +0000 UTC m=+22.727818471" watchObservedRunningTime="2026-07-28 17:09:43.845776448 +0000 UTC m=+22.728183600"
Jul 28 17:09:43.861858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:43.861812    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7n85v" podStartSLOduration=1.244269773 podStartE2EDuration="21.861799208s" totalImagesPullingTime="20.617529435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:43.861062173 +0000 UTC m=+22.743469287" watchObservedRunningTime="2026-07-28 17:09:43.861799208 +0000 UTC m=+22.744206325"
Jul 28 17:09:44.822816 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.822574    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="6d99d2e4958e11e8d0a034927e1f45908db898896ad7a033d0b21824eaddf49a" exitCode=0
Jul 28 17:09:44.823501 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.822894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"6d99d2e4958e11e8d0a034927e1f45908db898896ad7a033d0b21824eaddf49a"}
Jul 28 17:09:44.825336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.825313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vvb2g" event={"ID":"15a784d6-4b9f-48f7-b7af-95afd4bcfce4","Type":"ContainerStarted","Data":"d1ab0efa3f54a1b978e598776816c25b782f41b56128f7d09c63d9a051dc077c"}
Jul 28 17:09:44.826920 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.826899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x" event={"ID":"3eff3acd-73cb-48ec-bf7c-332fe083b72d","Type":"ContainerStarted","Data":"e3874dd3f9ec38c24e85ece2bd84fc2dd95894da344aaa3aa59f9e1e81d604fa"}
Jul 28 17:09:44.828975 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.828480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pzwwh" event={"ID":"99abb92a-6ff6-4307-af4f-7f8b0fa3674b","Type":"ContainerStarted","Data":"e9eb4c811bfaf2bb5b6db264a76bd559582547b286a1f475f1fa6d92e13f4a5b"}
Jul 28 17:09:44.831422 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.831397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"917839eb2a465bf11030a103121550437dfc8f8c0c8f76572f2ebe6e0daead27"}
Jul 28 17:09:44.831519 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.831460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"c74bbe45889b536063d17251c6940e527c251c261655f5f0b9a7127c23c9d177"}
Jul 28 17:09:44.831519 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.831502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"807c0cf2a6725628288beda7b83c77547f37f021f1a96e3a106f9507ee57ac42"}
Jul 28 17:09:44.831633 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.831543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"87cd7f2cf915e943cd5b49791da8cba4056c819511f5e1c15785b56946635636"}
Jul 28 17:09:44.831633 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.831583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"218b0010885f40fa8b70cc3371d522270494169ae7cf1e85bdeea3de59ed2161"}
Jul 28 17:09:44.833022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.833001    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fec44b9d7dbc7dbb1f61524a3eec673" containerID="0edbecee1bca77fcee150b7712bd8699d116ff02453a2839202fa0495001c5cf" exitCode=0
Jul 28 17:09:44.833092 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.833059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal" event={"ID":"9fec44b9d7dbc7dbb1f61524a3eec673","Type":"ContainerDied","Data":"0edbecee1bca77fcee150b7712bd8699d116ff02453a2839202fa0495001c5cf"}
Jul 28 17:09:44.834722 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.834699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfbzv" event={"ID":"66cd0b5b-1bba-4a4b-9a37-79d8e241db87","Type":"ContainerStarted","Data":"9ef8800c1d641446e242c7d34ae3cdf2e8f29fbe7ab77942f1efdc9b863d9b30"}
Jul 28 17:09:44.877202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.877110    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bfbzv" podStartSLOduration=2.349109652 podStartE2EDuration="22.877089859s" totalImagesPullingTime="20.527980207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:44.876502948 +0000 UTC m=+23.758910058" watchObservedRunningTime="2026-07-28 17:09:44.877089859 +0000 UTC m=+23.759496976"
Jul 28 17:09:44.892320 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:44.892263    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pzwwh" podStartSLOduration=3.902019791 podStartE2EDuration="22.892242685s" totalImagesPullingTime="18.990222894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:44.892082273 +0000 UTC m=+23.774489390" watchObservedRunningTime="2026-07-28 17:09:44.892242685 +0000 UTC m=+23.774649800"
Jul 28 17:09:45.652605 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.652583    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:09:45.667170 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.667142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:45.667285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.667190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:45.667285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.667143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:45.667285 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:45.667271    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:45.667479 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:45.667367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:45.667479 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:45.667418    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:45.840633 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.840596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"772aa0b0115a513deda3d134ba5926051e5c5e755e68e7864a82c9a3650e7acf"}
Jul 28 17:09:45.842713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.842662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal" event={"ID":"9fec44b9d7dbc7dbb1f61524a3eec673","Type":"ContainerStarted","Data":"1f2a89942ff7a005524a3bf3bf2cb1d11cf594f1703e08e821864026e0a2e919"}
Jul 28 17:09:45.844557 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.844532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2qcw" event={"ID":"f5af3a59-9f62-4254-b97e-ba7486e1cf54","Type":"ContainerStarted","Data":"082a1e12456fd7e318aa4b72b562e711dbd4085c7060da9c8fbe99d45a0b8ecc"}
Jul 28 17:09:45.846650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.846630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x" event={"ID":"3eff3acd-73cb-48ec-bf7c-332fe083b72d","Type":"ContainerStarted","Data":"ce96688c0b25c3b431c898e45f4c5f47a01b420e4d43d59fd764c08c7d8adbe0"}
Jul 28 17:09:45.858493 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.858449    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-17.ec2.internal" podStartSLOduration=22.858432984 podStartE2EDuration="22.858432984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.8582624 +0000 UTC m=+24.740669518" watchObservedRunningTime="2026-07-28 17:09:45.858432984 +0000 UTC m=+24.740840103"
Jul 28 17:09:45.858768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.858739    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vvb2g" podStartSLOduration=3.255089821 podStartE2EDuration="23.858732s" totalImagesPullingTime="20.603642179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:44.9073796 +0000 UTC m=+23.789786717" watchObservedRunningTime="2026-07-28 17:09:45.858732 +0000 UTC m=+24.741139115"
Jul 28 17:09:45.873344 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:45.873297    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k2qcw" podStartSLOduration=3.339687452 podStartE2EDuration="23.873283526s" totalImagesPullingTime="20.533596074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.872647835 +0000 UTC m=+24.755054974" watchObservedRunningTime="2026-07-28 17:09:45.873283526 +0000 UTC m=+24.755690642"
Jul 28 17:09:46.543707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:46.543594    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:45.652603373Z","UUID":"527c5cfb-1f28-446f-8722-1d2375a6f585","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:09:46.546099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:46.546074    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:09:46.546237 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:46.546122    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:09:47.667295 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.667125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:47.667847 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.667125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:47.667847 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:47.667393    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:47.667847 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.667125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:47.667847 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:47.667446    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:47.667847 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:47.667577    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:47.853849 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.853814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x" event={"ID":"3eff3acd-73cb-48ec-bf7c-332fe083b72d","Type":"ContainerStarted","Data":"db0c49019858a5d15ae468ba36d6f9f95bb2df450a2d75987514228dbddefbe0"}
Jul 28 17:09:47.857425 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.857395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"8b47e2cc093580a871f500f4048d1f466530747f04ce259818cb4e4aa2813113"}
Jul 28 17:09:47.871721 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:47.871673    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fj2x" podStartSLOduration=2.738365781 podStartE2EDuration="25.871653917s" totalImagesPullingTime="23.133288136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:47.871533452 +0000 UTC m=+26.753940568" watchObservedRunningTime="2026-07-28 17:09:47.871653917 +0000 UTC m=+26.754061034"
Jul 28 17:09:48.074006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:48.073978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:48.074653 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:48.074633    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:49.667403 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.667186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:49.668025 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.667186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:49.668025 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:49.667445    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:49.668025 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.667199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:49.668025 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:49.667509    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:49.668025 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:49.667581    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:49.862693 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.862658    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="800d0bd9b2eccffa76d8e75494ec19d9b126ecc256f11901a3cbf3f98634de89" exitCode=0
Jul 28 17:09:49.862916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.862747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"800d0bd9b2eccffa76d8e75494ec19d9b126ecc256f11901a3cbf3f98634de89"}
Jul 28 17:09:49.866406 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.866384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" event={"ID":"da4dbe31-ef2d-4a5f-a645-29079110aa50","Type":"ContainerStarted","Data":"bdee3a050354cf31a355c03b1be1d26ac4dc0f870d126052f59e0acd942aa74c"}
Jul 28 17:09:49.866715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.866690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:49.866821 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.866758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:49.866821 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.866797    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:49.881623 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.881604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:49.881725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.881690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:09:49.949880 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:49.949768    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk" podStartSLOduration=7.027448804 podStartE2EDuration="27.949747496s" totalImagesPullingTime="20.922298692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:49.949265417 +0000 UTC m=+28.831672523" watchObservedRunningTime="2026-07-28 17:09:49.949747496 +0000 UTC m=+28.832154611"
Jul 28 17:09:51.127688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.127656    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bztwt"]
Jul 28 17:09:51.128094 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.127829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:51.128094 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:51.127979    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:51.130472 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.130448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jtqn7"]
Jul 28 17:09:51.130595 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.130562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:51.130687 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:51.130665    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:51.131171 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.131152    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zk8h7"]
Jul 28 17:09:51.131262 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.131248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:51.131352 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:51.131334    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:51.299299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.299216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:51.299456 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.299335    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:09:51.299991 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.299953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pzwwh"
Jul 28 17:09:51.873919 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.873884    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="5ea2452c20ad437d8d28d30d01faa0650ac506ab5f5edec2bfb2f91f1453ba33" exitCode=0
Jul 28 17:09:51.874077 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:51.873965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"5ea2452c20ad437d8d28d30d01faa0650ac506ab5f5edec2bfb2f91f1453ba33"}
Jul 28 17:09:52.666864 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:52.666835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:52.666864 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:52.666843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:52.667343 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:52.666843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:52.667343 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:52.666966    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:52.667343 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:52.667072    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:52.667343 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:52.667145    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:52.878642 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:52.878474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerStarted","Data":"28efe4cd21de78b9142ff57cdde2aa21941212f4638bcaa4f47464dc822d2221"}
Jul 28 17:09:53.883232 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:53.883197    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="28efe4cd21de78b9142ff57cdde2aa21941212f4638bcaa4f47464dc822d2221" exitCode=0
Jul 28 17:09:53.883611 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:53.883249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"28efe4cd21de78b9142ff57cdde2aa21941212f4638bcaa4f47464dc822d2221"}
Jul 28 17:09:54.237772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.237736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:54.237954 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.237861    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:54.237954 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.237927    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret podName:bea06bea-ce5d-42a6-ad14-3025bee0c0a7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.237912965 +0000 UTC m=+65.120320060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret") pod "global-pull-secret-syncer-bztwt" (UID: "bea06bea-ce5d-42a6-ad14-3025bee0c0a7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:54.338102 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.338065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:54.338279 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.338217    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:54.338351 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.338292    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs podName:482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.338270147 +0000 UTC m=+65.220677240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs") pod "network-metrics-daemon-zk8h7" (UID: "482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:54.438637 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.438601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:54.438789 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.438756    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:54.438789 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.438776    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:54.438789 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.438785    2559 projected.go:196] Error preparing data for projected volume kube-api-access-h6jm2 for pod openshift-network-diagnostics/network-check-target-jtqn7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:54.438931 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.438836    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2 podName:e663d17a-7350-4a83-9111-8d313abeb23f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.438822827 +0000 UTC m=+65.321229921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h6jm2" (UniqueName: "kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2") pod "network-check-target-jtqn7" (UID: "e663d17a-7350-4a83-9111-8d313abeb23f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:54.666691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.666659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:54.666887 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.666730    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:54.666887 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:54.666835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:54.666887 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.666852    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:54.667032 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.666967    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:54.667168 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:54.667054    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:56.667081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:56.667044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:56.667654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:56.667093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:56.667654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:56.667093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:56.667654 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:56.667174    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jtqn7" podUID="e663d17a-7350-4a83-9111-8d313abeb23f"
Jul 28 17:09:56.667654 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:56.667273    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zk8h7" podUID="482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c"
Jul 28 17:09:56.667654 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:56.667342    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bztwt" podUID="bea06bea-ce5d-42a6-ad14-3025bee0c0a7"
Jul 28 17:09:57.396807 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.396777    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-17.ec2.internal" event="NodeReady"
Jul 28 17:09:57.396999 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.396987    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:09:57.433771 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.433731    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:09:57.436109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.436087    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-sr6tx"]
Jul 28 17:09:57.436257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.436230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.438124 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.438051    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"]
Jul 28 17:09:57.438333 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.438221    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.440563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.439690    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:09:57.440563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.439955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6jkxz\""
Jul 28 17:09:57.440563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.440132    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:09:57.440795 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.440753    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6mmmg\""
Jul 28 17:09:57.441097 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.441074    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"]
Jul 28 17:09:57.441241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.441224    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.441514 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.441345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 17:09:57.442447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.442424    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.443064 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.442897    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:09:57.443064 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.443029    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"]
Jul 28 17:09:57.443236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.443166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.444620 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.444600    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8mmmk"]
Jul 28 17:09:57.444754 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.444738    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"
Jul 28 17:09:57.445261 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.445243    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b4zkw\""
Jul 28 17:09:57.446226 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.446060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 17:09:57.446226 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.446134    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.446369 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.446350    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"]
Jul 28 17:09:57.446470 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.446449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.447388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.447372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 17:09:57.447619 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.447602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.448261 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.448238    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"]
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.448643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.448653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.448803    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.448934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449031    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.449496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 17:09:57.449839 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449610    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.449839 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449747    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6nn7c\""
Jul 28 17:09:57.449839 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449794    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66cf49fbb7-drwbq"]
Jul 28 17:09:57.450013 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449921    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.450013 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.449795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 17:09:57.450160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.450138    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-44fg7\""
Jul 28 17:09:57.452148 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.452130    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.453117 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.453098    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 17:09:57.453222 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.453131    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.453383 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.453366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 17:09:57.453825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.453805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-sr6tx"]
Jul 28 17:09:57.453941 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.453930    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.456121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.454861    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.456121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.455149    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.456121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.455436    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.456121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.455509    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 17:09:57.457167 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.456733    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pw77r\""
Jul 28 17:09:57.457167 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.457014    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bbvqb\""
Jul 28 17:09:57.457979 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.457959    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:09:57.458453 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.458134    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.458453 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.458284    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:09:57.458631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.458534    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.460763 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.460743    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9bmhj\""
Jul 28 17:09:57.467332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.463879    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rqz7s\""
Jul 28 17:09:57.467332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.465000    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 17:09:57.468594 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.468571    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.469135 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.469111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:09:57.469272 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.469250    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.469377 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.469255    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:09:57.469506 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.469262    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 17:09:57.470842 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.470312    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:09:57.471680 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.471383    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"]
Jul 28 17:09:57.472170 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.472151    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 17:09:57.472469 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.472293    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"]
Jul 28 17:09:57.473997 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.473963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"]
Jul 28 17:09:57.474290 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.474269    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"]
Jul 28 17:09:57.474945 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.474923    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 17:09:57.475544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.475222    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"]
Jul 28 17:09:57.478282 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.478266    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"]
Jul 28 17:09:57.478407 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.478385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.480477 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480456    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:09:57.480577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8mmmk"]
Jul 28 17:09:57.480577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66cf49fbb7-drwbq"]
Jul 28 17:09:57.480577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480531    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tstvj"]
Jul 28 17:09:57.480577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"
Jul 28 17:09:57.480785 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.480753    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 17:09:57.481083 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.481065    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 17:09:57.481083 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.481078    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9lttv\""
Jul 28 17:09:57.483081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.483063    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.483310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.483182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fjdbs\""
Jul 28 17:09:57.483403 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.483263    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.483655 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.483638    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n2zb8"]
Jul 28 17:09:57.483825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.483729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.486438 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.486422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"]
Jul 28 17:09:57.486525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.486467    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"]
Jul 28 17:09:57.486525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.486495    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"]
Jul 28 17:09:57.486525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.486512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tstvj"]
Jul 28 17:09:57.486682 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.486593    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.487159 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.487138    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:09:57.487255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.487175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8842r\""
Jul 28 17:09:57.487255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.487193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:09:57.488525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.488507    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:09:57.489089 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.489071    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q4trs\""
Jul 28 17:09:57.489317 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.488893    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:09:57.489408 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.489392    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n2zb8"]
Jul 28 17:09:57.489483 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.488891    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:09:57.565356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.565356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d136840f-e10d-4846-9f16-ffbc2ae86c70-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.565356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t8v\" (UniqueName: \"kubernetes.io/projected/d136840f-e10d-4846-9f16-ffbc2ae86c70-kube-api-access-p7t8v\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76dfb982-e9ff-4d68-9bcd-339a79c054db-serving-cert\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8lv\" (UniqueName: \"kubernetes.io/projected/17e5be93-25fe-40b8-a784-ba4e359a1f50-kube-api-access-kf8lv\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-default-certificate\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdl9\" (UniqueName: \"kubernetes.io/projected/ea489c3b-940c-4ca9-8b3b-111602e74186-kube-api-access-6mdl9\") pod \"network-check-source-7d88cb55d8-q5m5c\" (UID: \"ea489c3b-940c-4ca9-8b3b-111602e74186\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/024d38b7-c1c9-4e99-847d-5d65c15530d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.565634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wl6\" (UniqueName: \"kubernetes.io/projected/d142323f-846b-471d-82b6-5dfbed8e5128-kube-api-access-95wl6\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-stats-auth\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-tmp\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-config\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-snapshots\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76dfb982-e9ff-4d68-9bcd-339a79c054db-config\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-serving-cert\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkntl\" (UniqueName: \"kubernetes.io/projected/d3db7855-e235-4278-9b62-79e43b6fcb11-kube-api-access-vkntl\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.565982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrqh\" (UniqueName: \"kubernetes.io/projected/ac10475d-cc6b-4df9-894e-2d750adb3c81-kube-api-access-qzrqh\") pod \"volume-data-source-validator-549c6d54cf-bt9qs\" (UID: \"ac10475d-cc6b-4df9-894e-2d750adb3c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.566081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-service-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566088    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c594f79-fcb8-4590-bf2f-b4991d614eb8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-trusted-ca\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44cq7\" (UniqueName: \"kubernetes.io/projected/76dfb982-e9ff-4d68-9bcd-339a79c054db-kube-api-access-44cq7\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c594f79-fcb8-4590-bf2f-b4991d614eb8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6vs\" (UniqueName: \"kubernetes.io/projected/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-kube-api-access-lk6vs\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzqd\" (UniqueName: \"kubernetes.io/projected/6c594f79-fcb8-4590-bf2f-b4991d614eb8-kube-api-access-trzqd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17e5be93-25fe-40b8-a784-ba4e359a1f50-serving-cert\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mgl\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.566671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.566552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.667256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/024d38b7-c1c9-4e99-847d-5d65c15530d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wl6\" (UniqueName: \"kubernetes.io/projected/d142323f-846b-471d-82b6-5dfbed8e5128-kube-api-access-95wl6\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffr5c\" (UniqueName: \"kubernetes.io/projected/ddadacb7-1b96-4c8b-8e13-9f2043680423-kube-api-access-ffr5c\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-stats-auth\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-tmp\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-config\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-snapshots\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddadacb7-1b96-4c8b-8e13-9f2043680423-cert\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76dfb982-e9ff-4d68-9bcd-339a79c054db-config\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-serving-cert\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb53b3e-7d01-4584-b233-2bb144f857be-config-volume\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkntl\" (UniqueName: \"kubernetes.io/projected/d3db7855-e235-4278-9b62-79e43b6fcb11-kube-api-access-vkntl\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.667713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrqh\" (UniqueName: \"kubernetes.io/projected/ac10475d-cc6b-4df9-894e-2d750adb3c81-kube-api-access-qzrqh\") pod \"volume-data-source-validator-549c6d54cf-bt9qs\" (UID: \"ac10475d-cc6b-4df9-894e-2d750adb3c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.667697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.667801    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.667886    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.167850299 +0000 UTC m=+37.050257394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.668418    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.168400858 +0000 UTC m=+37.050807974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-service-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/024d38b7-c1c9-4e99-847d-5d65c15530d7-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.668496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-tmp\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17e5be93-25fe-40b8-a784-ba4e359a1f50-snapshots\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-config\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.668764    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:09:57.668913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c594f79-fcb8-4590-bf2f-b4991d614eb8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-service-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.668960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76dfb982-e9ff-4d68-9bcd-339a79c054db-config\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669024    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.169008028 +0000 UTC m=+37.051415142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-trusted-ca\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.669175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44cq7\" (UniqueName: \"kubernetes.io/projected/76dfb982-e9ff-4d68-9bcd-339a79c054db-kube-api-access-44cq7\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c594f79-fcb8-4590-bf2f-b4991d614eb8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6vs\" (UniqueName: \"kubernetes.io/projected/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-kube-api-access-lk6vs\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669276    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669289    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669330    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.16931662 +0000 UTC m=+37.051723734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzqd\" (UniqueName: \"kubernetes.io/projected/6c594f79-fcb8-4590-bf2f-b4991d614eb8-kube-api-access-trzqd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.669394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17e5be93-25fe-40b8-a784-ba4e359a1f50-serving-cert\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mgl\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb53b3e-7d01-4584-b233-2bb144f857be-tmp-dir\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d136840f-e10d-4846-9f16-ffbc2ae86c70-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t8v\" (UniqueName: \"kubernetes.io/projected/d136840f-e10d-4846-9f16-ffbc2ae86c70-kube-api-access-p7t8v\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76dfb982-e9ff-4d68-9bcd-339a79c054db-serving-cert\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8lv\" (UniqueName: \"kubernetes.io/projected/17e5be93-25fe-40b8-a784-ba4e359a1f50-kube-api-access-kf8lv\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-default-certificate\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669671    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:09:57.669725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdl9\" (UniqueName: \"kubernetes.io/projected/ea489c3b-940c-4ca9-8b3b-111602e74186-kube-api-access-6mdl9\") pod \"network-check-source-7d88cb55d8-q5m5c\" (UID: \"ea489c3b-940c-4ca9-8b3b-111602e74186\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"
Jul 28 17:09:57.670302 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.669740    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.169727026 +0000 UTC m=+37.052134124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:09:57.670302 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzb6\" (UniqueName: \"kubernetes.io/projected/ccb53b3e-7d01-4584-b233-2bb144f857be-kube-api-access-rqzb6\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.670302 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.670302 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.669857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-trusted-ca\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.672568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.670555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e5be93-25fe-40b8-a784-ba4e359a1f50-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.672568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.670938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c594f79-fcb8-4590-bf2f-b4991d614eb8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.672814    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.672883    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.172850711 +0000 UTC m=+37.055257810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.672858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d136840f-e10d-4846-9f16-ffbc2ae86c70-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.673118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.673419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-serving-cert\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.673424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c594f79-fcb8-4590-bf2f-b4991d614eb8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.673494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.674758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.674270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-stats-auth\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.675351 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.674815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.675351 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.674928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17e5be93-25fe-40b8-a784-ba4e359a1f50-serving-cert\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.675719 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.675699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-default-certificate\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.675800 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.675753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76dfb982-e9ff-4d68-9bcd-339a79c054db-serving-cert\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.678592 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.678569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkntl\" (UniqueName: \"kubernetes.io/projected/d3db7855-e235-4278-9b62-79e43b6fcb11-kube-api-access-vkntl\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:57.679245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.679199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wl6\" (UniqueName: \"kubernetes.io/projected/d142323f-846b-471d-82b6-5dfbed8e5128-kube-api-access-95wl6\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:57.682609 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.682319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrqh\" (UniqueName: \"kubernetes.io/projected/ac10475d-cc6b-4df9-894e-2d750adb3c81-kube-api-access-qzrqh\") pod \"volume-data-source-validator-549c6d54cf-bt9qs\" (UID: \"ac10475d-cc6b-4df9-894e-2d750adb3c81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"
Jul 28 17:09:57.682609 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.682563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8lv\" (UniqueName: \"kubernetes.io/projected/17e5be93-25fe-40b8-a784-ba4e359a1f50-kube-api-access-kf8lv\") pod \"insights-operator-5885fdb759-sr6tx\" (UID: \"17e5be93-25fe-40b8-a784-ba4e359a1f50\") " pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.683371 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.682985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t8v\" (UniqueName: \"kubernetes.io/projected/d136840f-e10d-4846-9f16-ffbc2ae86c70-kube-api-access-p7t8v\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:57.683371 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.683313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44cq7\" (UniqueName: \"kubernetes.io/projected/76dfb982-e9ff-4d68-9bcd-339a79c054db-kube-api-access-44cq7\") pod \"service-ca-operator-5d6c75f84f-nmw2j\" (UID: \"76dfb982-e9ff-4d68-9bcd-339a79c054db\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.683509 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.683373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.683574 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.683551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdl9\" (UniqueName: \"kubernetes.io/projected/ea489c3b-940c-4ca9-8b3b-111602e74186-kube-api-access-6mdl9\") pod \"network-check-source-7d88cb55d8-q5m5c\" (UID: \"ea489c3b-940c-4ca9-8b3b-111602e74186\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"
Jul 28 17:09:57.683785 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.683762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzqd\" (UniqueName: \"kubernetes.io/projected/6c594f79-fcb8-4590-bf2f-b4991d614eb8-kube-api-access-trzqd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k\" (UID: \"6c594f79-fcb8-4590-bf2f-b4991d614eb8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.684707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.684658    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6vs\" (UniqueName: \"kubernetes.io/projected/5cd5f7ef-ed83-49d5-86c2-7748c28fde5f-kube-api-access-lk6vs\") pod \"console-operator-584dcb6b54-8mmmk\" (UID: \"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f\") " pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.684981 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.684963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mgl\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:57.763124 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.763095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-sr6tx"
Jul 28 17:09:57.770939 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.770917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzb6\" (UniqueName: \"kubernetes.io/projected/ccb53b3e-7d01-4584-b233-2bb144f857be-kube-api-access-rqzb6\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.771086 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.770946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.771086 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.770992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffr5c\" (UniqueName: \"kubernetes.io/projected/ddadacb7-1b96-4c8b-8e13-9f2043680423-kube-api-access-ffr5c\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.771086 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.771072    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:57.771245 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:57.771137    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:09:58.271117603 +0000 UTC m=+37.153524697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:57.771245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.771159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddadacb7-1b96-4c8b-8e13-9f2043680423-cert\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.771245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.771201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb53b3e-7d01-4584-b233-2bb144f857be-config-volume\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.771410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.771314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb53b3e-7d01-4584-b233-2bb144f857be-tmp-dir\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.771608 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.771589    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb53b3e-7d01-4584-b233-2bb144f857be-tmp-dir\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.771759 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.771737    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb53b3e-7d01-4584-b233-2bb144f857be-config-volume\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.773689 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.773654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ddadacb7-1b96-4c8b-8e13-9f2043680423-cert\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.781692 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.781670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzb6\" (UniqueName: \"kubernetes.io/projected/ccb53b3e-7d01-4584-b233-2bb144f857be-kube-api-access-rqzb6\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:57.781911 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.781890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffr5c\" (UniqueName: \"kubernetes.io/projected/ddadacb7-1b96-4c8b-8e13-9f2043680423-kube-api-access-ffr5c\") pod \"ingress-canary-n2zb8\" (UID: \"ddadacb7-1b96-4c8b-8e13-9f2043680423\") " pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:57.796854 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.796832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"
Jul 28 17:09:57.803575 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.803557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"
Jul 28 17:09:57.810980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.810960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:09:57.823723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.823678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"
Jul 28 17:09:57.839357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.839336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"
Jul 28 17:09:57.852948 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:57.852927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n2zb8"
Jul 28 17:09:58.171361 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.171116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k"]
Jul 28 17:09:58.174409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:58.174544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:58.174639 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:58.174921 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:58.174921 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:58.174921 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.174809    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:09:58.174921 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.174910    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.174971    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.174949043 +0000 UTC m=+38.057356139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175013    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.174985635 +0000 UTC m=+38.057392729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175035    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175086    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.175068263 +0000 UTC m=+38.057475363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175101    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175149    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.175138091 +0000 UTC m=+38.057545187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:09:58.175166 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.174815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:58.175499 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.174815    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:09:58.175499 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175322    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:09:58.175499 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175372    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.175358952 +0000 UTC m=+38.057766081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:09:58.175640 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.175559    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.17553987 +0000 UTC m=+38.057946963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:09:58.183805 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.183780    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-8mmmk"]
Jul 28 17:09:58.194971 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.194948    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c"]
Jul 28 17:09:58.202974 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.202941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n2zb8"]
Jul 28 17:09:58.203815 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.203801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs"]
Jul 28 17:09:58.204591 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.204574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-sr6tx"]
Jul 28 17:09:58.208702 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.208681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j"]
Jul 28 17:09:58.211768 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:58.211748    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e5be93_25fe_40b8_a784_ba4e359a1f50.slice/crio-7452b95a1db94119556482b16da522bbf5805a7aabc05a6ca11e01d6a6a09dd4 WatchSource:0}: Error finding container 7452b95a1db94119556482b16da522bbf5805a7aabc05a6ca11e01d6a6a09dd4: Status 404 returned error can't find the container with id 7452b95a1db94119556482b16da522bbf5805a7aabc05a6ca11e01d6a6a09dd4
Jul 28 17:09:58.213010 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:58.212984    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c594f79_fcb8_4590_bf2f_b4991d614eb8.slice/crio-fdbc08c7499ed6dc9a10a802a293bf5089aa6d80a34eeb66e84a0d80039cb183 WatchSource:0}: Error finding container fdbc08c7499ed6dc9a10a802a293bf5089aa6d80a34eeb66e84a0d80039cb183: Status 404 returned error can't find the container with id fdbc08c7499ed6dc9a10a802a293bf5089aa6d80a34eeb66e84a0d80039cb183
Jul 28 17:09:58.214033 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:09:58.213999    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac10475d_cc6b_4df9_894e_2d750adb3c81.slice/crio-dd172bedcd48fcc10c02bc80d3a3858d74015b1c8cf5c31935e6ff5214781307 WatchSource:0}: Error finding container dd172bedcd48fcc10c02bc80d3a3858d74015b1c8cf5c31935e6ff5214781307: Status 404 returned error can't find the container with id dd172bedcd48fcc10c02bc80d3a3858d74015b1c8cf5c31935e6ff5214781307
Jul 28 17:09:58.275833 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.275809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:58.275982 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.275965    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:58.276057 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:58.276026    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:09:59.276011884 +0000 UTC m=+38.158418977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:58.667194 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.667160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:09:58.668217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.667497    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:09:58.668217 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.667503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:09:58.670132 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.670088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:09:58.670636 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.670309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:09:58.670636 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.670399    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nvllg\""
Jul 28 17:09:58.670636 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.670507    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mz7f2\""
Jul 28 17:09:58.901604 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.901567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-sr6tx" event={"ID":"17e5be93-25fe-40b8-a784-ba4e359a1f50","Type":"ContainerStarted","Data":"7452b95a1db94119556482b16da522bbf5805a7aabc05a6ca11e01d6a6a09dd4"}
Jul 28 17:09:58.910250 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.910157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" event={"ID":"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f","Type":"ContainerStarted","Data":"504b12e16e03c1e8b390fe7ea99ad26a076b534730e84fa41938f19f1adce8dd"}
Jul 28 17:09:58.916627 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.915630    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="3f796e76d3be745e98eef4f8c6f0b95d364d95459918825f5e132733c1afe7e8" exitCode=0
Jul 28 17:09:58.916627 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.915711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"3f796e76d3be745e98eef4f8c6f0b95d364d95459918825f5e132733c1afe7e8"}
Jul 28 17:09:58.923411 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.923357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c" event={"ID":"ea489c3b-940c-4ca9-8b3b-111602e74186","Type":"ContainerStarted","Data":"da04fe437274504c61b94e779cca16e3c38d3dc8987c1ebacdcf08ec684ae11e"}
Jul 28 17:09:58.926883 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.926776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs" event={"ID":"ac10475d-cc6b-4df9-894e-2d750adb3c81","Type":"ContainerStarted","Data":"dd172bedcd48fcc10c02bc80d3a3858d74015b1c8cf5c31935e6ff5214781307"}
Jul 28 17:09:58.929336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.929247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n2zb8" event={"ID":"ddadacb7-1b96-4c8b-8e13-9f2043680423","Type":"ContainerStarted","Data":"53fa7182e4c04c4f8b1527c70b7ffef5cade3b135f6b54f6ab2ff5f376a02a8f"}
Jul 28 17:09:58.933584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.933529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j" event={"ID":"76dfb982-e9ff-4d68-9bcd-339a79c054db","Type":"ContainerStarted","Data":"8ecb2a008f595cbab2cc551d00f7906662fe3bfa93bdb09db73a8eef21c2e1fd"}
Jul 28 17:09:58.936130 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:58.936040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k" event={"ID":"6c594f79-fcb8-4590-bf2f-b4991d614eb8","Type":"ContainerStarted","Data":"fdbc08c7499ed6dc9a10a802a293bf5089aa6d80a34eeb66e84a0d80039cb183"}
Jul 28 17:09:59.185464 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:59.185464 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:09:59.185679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:09:59.185679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:09:59.185679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:09:59.185679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.185569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:09:59.185948 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.185764    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:59.185948 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.185833    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.185814935 +0000 UTC m=+40.068222043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:09:59.185948 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.185923    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:09:59.186115 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.185962    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.185950561 +0000 UTC m=+40.068357655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:09:59.186115 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186025    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:09:59.186115 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186063    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.186050155 +0000 UTC m=+40.068457252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:09:59.186258 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186145    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.18613461 +0000 UTC m=+40.068541721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:09:59.186258 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186220    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:09:59.186258 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186233    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:09:59.186413 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186263    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.18625308 +0000 UTC m=+40.068660179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:09:59.186413 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186325    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:09:59.186413 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.186364    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.186346998 +0000 UTC m=+40.068754094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:09:59.286284 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.286248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:09:59.286464 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.286449    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:59.286539 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:09:59.286525    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:10:01.286505739 +0000 UTC m=+40.168912839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:59.949441 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.949132    2559 generic.go:356] "Generic (PLEG): container finished" podID="dab1330b-69d5-4f1c-9266-1ae217f6b1c7" containerID="69615677f0e39f436c487fa3196d28bb20d0a0ff0a26199c68f3b0adbc64eb93" exitCode=0
Jul 28 17:09:59.949441 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:09:59.949202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerDied","Data":"69615677f0e39f436c487fa3196d28bb20d0a0ff0a26199c68f3b0adbc64eb93"}
Jul 28 17:10:00.968740 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:00.968449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" event={"ID":"dab1330b-69d5-4f1c-9266-1ae217f6b1c7","Type":"ContainerStarted","Data":"e923cba28a1d0b63271142b1dfa7a02f094baf614a1e043711836bf00f33c4d4"}
Jul 28 17:10:00.993426 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:00.993369    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w8gdg" podStartSLOduration=7.935861417 podStartE2EDuration="38.993348441s" totalImagesPullingTime="31.057487024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:00.992215163 +0000 UTC m=+39.874622280" watchObservedRunningTime="2026-07-28 17:10:00.993348441 +0000 UTC m=+39.875755550"
Jul 28 17:10:01.208105 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:01.208105 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208251    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.208261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208306    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208329    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208309    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208355    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:10:01.208380 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208359    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208321    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208305625 +0000 UTC m=+44.090712718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208428    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208416072 +0000 UTC m=+44.090823170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208441    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208435177 +0000 UTC m=+44.090842271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208454    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208447706 +0000 UTC m=+44.090854800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208470    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208465625 +0000 UTC m=+44.090872719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:10:01.208703 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.208483    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.208479127 +0000 UTC m=+44.090886220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:10:01.309922 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:01.309892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:01.310088 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.310071    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:01.310149 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:01.310139    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:10:05.310119581 +0000 UTC m=+44.192526678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:05.248131 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248277    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248304    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248348    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248358    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248309    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248347963 +0000 UTC m=+52.130755057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248434    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248418195 +0000 UTC m=+52.130825292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248463    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248450446 +0000 UTC m=+52.130857546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248485    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248477277 +0000 UTC m=+52.130884377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.248580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:05.248691 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248679    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:10:05.249503 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248692    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248680525 +0000 UTC m=+52.131087626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:10:05.249503 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.248723    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.248706395 +0000 UTC m=+52.131113492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:10:05.349645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:05.349597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:05.349830 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.349767    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:05.349906 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:05.349839    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:10:13.349823933 +0000 UTC m=+52.232231027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:06.985336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.985093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c" event={"ID":"ea489c3b-940c-4ca9-8b3b-111602e74186","Type":"ContainerStarted","Data":"d02bcac8d0d9152df5e272d4af3879201347b456c625a85d00488ffc021e44b3"}
Jul 28 17:10:06.987853 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.987827    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs" event={"ID":"ac10475d-cc6b-4df9-894e-2d750adb3c81","Type":"ContainerStarted","Data":"66acf8b5fe74827f84fd6b5b7f6f1b30663fa7063a25f4af3ea8f0018fc335e6"}
Jul 28 17:10:06.989041 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.989016    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n2zb8" event={"ID":"ddadacb7-1b96-4c8b-8e13-9f2043680423","Type":"ContainerStarted","Data":"eedb949fc2a969d2c149db2590ef729402ff38679cc7ab7f6ea6ac66c85670d9"}
Jul 28 17:10:06.990645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.990619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j" event={"ID":"76dfb982-e9ff-4d68-9bcd-339a79c054db","Type":"ContainerStarted","Data":"fe88da23570a543552f28a177bc9db6dab611b6cf802848316c1d20a3f658958"}
Jul 28 17:10:06.992186 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.992163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k" event={"ID":"6c594f79-fcb8-4590-bf2f-b4991d614eb8","Type":"ContainerStarted","Data":"2e3b4731a8708299fb1d5bb19119f1dd96302eea663628cb6a2f0b1fda28f3b9"}
Jul 28 17:10:06.993727 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.993703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-sr6tx" event={"ID":"17e5be93-25fe-40b8-a784-ba4e359a1f50","Type":"ContainerStarted","Data":"8f533f215f716728433df7f764fedf79dc72cfe6c7619bcb73ae78db1c8c6410"}
Jul 28 17:10:06.995422 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.995406    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/0.log"
Jul 28 17:10:06.995517 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.995440    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd5f7ef-ed83-49d5-86c2-7748c28fde5f" containerID="7baee9e8d1fde99d48dbcfa410080a89bc74a7b566597673067cbf991097ddc8" exitCode=255
Jul 28 17:10:06.995517 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.995474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" event={"ID":"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f","Type":"ContainerDied","Data":"7baee9e8d1fde99d48dbcfa410080a89bc74a7b566597673067cbf991097ddc8"}
Jul 28 17:10:06.995701 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:06.995688    2559 scope.go:117] "RemoveContainer" containerID="7baee9e8d1fde99d48dbcfa410080a89bc74a7b566597673067cbf991097ddc8"
Jul 28 17:10:07.032663 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.032619    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bt9qs" podStartSLOduration=35.340960535 podStartE2EDuration="43.032601021s" totalImagesPullingTime="7.691640486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.030808034 +0000 UTC m=+45.913215149" watchObservedRunningTime="2026-07-28 17:10:07.032601021 +0000 UTC m=+45.915008140"
Jul 28 17:10:07.032814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.032790    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-q5m5c" podStartSLOduration=31.211560154 podStartE2EDuration="39.032782575s" totalImagesPullingTime="7.821222421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.008528071 +0000 UTC m=+45.890935187" watchObservedRunningTime="2026-07-28 17:10:07.032782575 +0000 UTC m=+45.915189695"
Jul 28 17:10:07.077803 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.077340    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j" podStartSLOduration=35.266410518 podStartE2EDuration="43.077319975s" totalImagesPullingTime="7.810909457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.053228063 +0000 UTC m=+45.935635179" watchObservedRunningTime="2026-07-28 17:10:07.077319975 +0000 UTC m=+45.959727088"
Jul 28 17:10:07.106422 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.106366    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-sr6tx" podStartSLOduration=35.288514863 podStartE2EDuration="43.106347006s" totalImagesPullingTime="7.817832143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.078645213 +0000 UTC m=+45.961052329" watchObservedRunningTime="2026-07-28 17:10:07.106347006 +0000 UTC m=+45.988754117"
Jul 28 17:10:07.156643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.156421    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n2zb8" podStartSLOduration=2.343457643 podStartE2EDuration="10.15639816s" totalImagesPullingTime="7.812940517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.132315204 +0000 UTC m=+46.014722322" watchObservedRunningTime="2026-07-28 17:10:07.15639816 +0000 UTC m=+46.038805277"
Jul 28 17:10:07.324477 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.324426    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k" podStartSLOduration=35.508902938 podStartE2EDuration="43.324406099s" totalImagesPullingTime="7.815503161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:07.156716634 +0000 UTC m=+46.039123751" watchObservedRunningTime="2026-07-28 17:10:07.324406099 +0000 UTC m=+46.206813215"
Jul 28 17:10:07.324817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.324803    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5gg8k"]
Jul 28 17:10:07.359060 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.359031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gg8k"]
Jul 28 17:10:07.359221 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.359185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.361650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.361618    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:10:07.361789 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.361652    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:10:07.362020 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.362004    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xm8pw\""
Jul 28 17:10:07.468626 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.468582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-crio-socket\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.468823 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.468663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkc8\" (UniqueName: \"kubernetes.io/projected/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-api-access-fmkc8\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.468823 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.468750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.468952 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.468885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-data-volume\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.468952 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.468907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-data-volume\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-crio-socket\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkc8\" (UniqueName: \"kubernetes.io/projected/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-api-access-fmkc8\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570252 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:07.570241    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:07.570448 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:07.570316    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:08.070292712 +0000 UTC m=+46.952699828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:07.570448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570422    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-data-volume\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-crio-socket\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.570248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.570659 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:07.570552    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:07.570659 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:07.570616    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:08.070599614 +0000 UTC m=+46.953006711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:07.582503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.582432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkc8\" (UniqueName: \"kubernetes.io/projected/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-api-access-fmkc8\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:07.811145 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.811111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:07.811288 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:07.811167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:08.000368 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.000295    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:10:08.000761 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.000746    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/0.log"
Jul 28 17:10:08.000815 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.000782    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd5f7ef-ed83-49d5-86c2-7748c28fde5f" containerID="380d915a52f8b19d8dabfe42452448a905287ae312e8c7ea6a5a1014e282c3c5" exitCode=255
Jul 28 17:10:08.000927 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.000901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" event={"ID":"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f","Type":"ContainerDied","Data":"380d915a52f8b19d8dabfe42452448a905287ae312e8c7ea6a5a1014e282c3c5"}
Jul 28 17:10:08.000990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.000964    2559 scope.go:117] "RemoveContainer" containerID="7baee9e8d1fde99d48dbcfa410080a89bc74a7b566597673067cbf991097ddc8"
Jul 28 17:10:08.001219 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.001198    2559 scope.go:117] "RemoveContainer" containerID="380d915a52f8b19d8dabfe42452448a905287ae312e8c7ea6a5a1014e282c3c5"
Jul 28 17:10:08.001421 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:08.001386    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8mmmk_openshift-console-operator(5cd5f7ef-ed83-49d5-86c2-7748c28fde5f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" podUID="5cd5f7ef-ed83-49d5-86c2-7748c28fde5f"
Jul 28 17:10:08.074379 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.074349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:08.074542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:08.074469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:08.074604 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:08.074468    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:08.074671 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:08.074627    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:09.0746036 +0000 UTC m=+47.957010697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:08.075148 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:08.075027    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:08.075148 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:08.075104    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:09.075075144 +0000 UTC m=+47.957482251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:09.005977 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:09.005950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:10:09.006371 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:09.006296    2559 scope.go:117] "RemoveContainer" containerID="380d915a52f8b19d8dabfe42452448a905287ae312e8c7ea6a5a1014e282c3c5"
Jul 28 17:10:09.006507 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:09.006485    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-8mmmk_openshift-console-operator(5cd5f7ef-ed83-49d5-86c2-7748c28fde5f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" podUID="5cd5f7ef-ed83-49d5-86c2-7748c28fde5f"
Jul 28 17:10:09.087461 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:09.087429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:09.087629 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:09.087524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:09.087713 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:09.087629    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:09.087770 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:09.087726    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:11.087705844 +0000 UTC m=+49.970112951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:09.087829 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:09.087783    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:09.087901 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:09.087847    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:11.087833792 +0000 UTC m=+49.970240890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:09.494800 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:09.494776    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vvb2g_15a784d6-4b9f-48f7-b7af-95afd4bcfce4/dns-node-resolver/0.log"
Jul 28 17:10:10.300155 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:10.300130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfbzv_66cd0b5b-1bba-4a4b-9a37-79d8e241db87/node-ca/0.log"
Jul 28 17:10:11.094431 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:11.094404    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n2zb8_ddadacb7-1b96-4c8b-8e13-9f2043680423/serve-healthcheck-canary/0.log"
Jul 28 17:10:11.105691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:11.105660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:11.105834 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:11.105769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:11.105937 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:11.105776    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:11.105989 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:11.105939    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:15.105918635 +0000 UTC m=+53.988325747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:11.105989 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:11.105834    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:11.106095 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:11.106001    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:15.105985839 +0000 UTC m=+53.988392932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:13.329567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:13.329567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329677    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.329681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329742    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.329721322 +0000 UTC m=+68.212128417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : secret "router-metrics-certs-default" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329757    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329795    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329802    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329819    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329810    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f76759cf4-rz9l8: secret "image-registry-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329799    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle podName:d142323f-846b-471d-82b6-5dfbed8e5128 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.329780514 +0000 UTC m=+68.212187624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle") pod "router-default-66cf49fbb7-drwbq" (UID: "d142323f-846b-471d-82b6-5dfbed8e5128") : configmap references non-existent config key: service-ca.crt
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329911    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls podName:4ced5784-9221-42e6-9012-315380e55baf nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.329894591 +0000 UTC m=+68.212301690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls") pod "image-registry-7f76759cf4-rz9l8" (UID: "4ced5784-9221-42e6-9012-315380e55baf") : secret "image-registry-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.329933    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls podName:d136840f-e10d-4846-9f16-ffbc2ae86c70 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.32992291 +0000 UTC m=+68.212330004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-njt9d" (UID: "d136840f-e10d-4846-9f16-ffbc2ae86c70") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:10:13.330099 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.330018    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls podName:d3db7855-e235-4278-9b62-79e43b6fcb11 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.330009676 +0000 UTC m=+68.212416772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls") pod "cluster-samples-operator-6748489656-97z85" (UID: "d3db7855-e235-4278-9b62-79e43b6fcb11") : secret "samples-operator-tls" not found
Jul 28 17:10:13.330575 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.330039    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert podName:024d38b7-c1c9-4e99-847d-5d65c15530d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.33003241 +0000 UTC m=+68.212439509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-8wdww" (UID: "024d38b7-c1c9-4e99-847d-5d65c15530d7") : secret "networking-console-plugin-cert" not found
Jul 28 17:10:13.430750 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:13.430727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:13.430900 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.430837    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:13.430900 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:13.430892    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls podName:ccb53b3e-7d01-4584-b233-2bb144f857be nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.430865934 +0000 UTC m=+68.313273028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls") pod "dns-default-tstvj" (UID: "ccb53b3e-7d01-4584-b233-2bb144f857be") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:15.144796 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:15.144762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:15.145248 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:15.144842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:15.145248 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:15.144954    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:15.145248 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:15.145010    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls podName:42dc4d4c-4a18-4939-99cf-85b7e1b368c3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:23.144996448 +0000 UTC m=+62.027403542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gg8k" (UID: "42dc4d4c-4a18-4939-99cf-85b7e1b368c3") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:15.145352 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:15.145286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:17.812095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:17.812060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:17.812608 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:17.812111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:17.812608 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:17.812464    2559 scope.go:117] "RemoveContainer" containerID="380d915a52f8b19d8dabfe42452448a905287ae312e8c7ea6a5a1014e282c3c5"
Jul 28 17:10:18.034513 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:18.034486    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:10:18.034667 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:18.034544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" event={"ID":"5cd5f7ef-ed83-49d5-86c2-7748c28fde5f","Type":"ContainerStarted","Data":"056d6dbadae2667d1af4e433dafcb6a6edcc16aa5c2c8668e7516641e04bc146"}
Jul 28 17:10:18.034804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:18.034786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:18.054782 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:18.054723    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk" podStartSLOduration=46.245367812 podStartE2EDuration="54.054685159s" totalImagesPullingTime="7.809317347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:18.054320813 +0000 UTC m=+56.936727977" watchObservedRunningTime="2026-07-28 17:10:18.054685159 +0000 UTC m=+56.937092276"
Jul 28 17:10:18.353580 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:18.353548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-8mmmk"
Jul 28 17:10:21.888664 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:21.888639    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pmvhk"
Jul 28 17:10:23.217054 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:23.217015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:23.219426 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:23.219401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42dc4d4c-4a18-4939-99cf-85b7e1b368c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gg8k\" (UID: \"42dc4d4c-4a18-4939-99cf-85b7e1b368c3\") " pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:23.272885 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:23.272837    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xm8pw\""
Jul 28 17:10:23.280707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:23.280688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gg8k"
Jul 28 17:10:23.430248 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:23.430219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gg8k"]
Jul 28 17:10:23.433959 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:23.433932    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dc4d4c_4a18_4939_99cf_85b7e1b368c3.slice/crio-97358730c60ede29ab2aaad6c6956a4a6fcd09e7868376950f8afd4362b5f034 WatchSource:0}: Error finding container 97358730c60ede29ab2aaad6c6956a4a6fcd09e7868376950f8afd4362b5f034: Status 404 returned error can't find the container with id 97358730c60ede29ab2aaad6c6956a4a6fcd09e7868376950f8afd4362b5f034
Jul 28 17:10:24.054990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:24.054947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gg8k" event={"ID":"42dc4d4c-4a18-4939-99cf-85b7e1b368c3","Type":"ContainerStarted","Data":"97358730c60ede29ab2aaad6c6956a4a6fcd09e7868376950f8afd4362b5f034"}
Jul 28 17:10:26.250085 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.250053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:10:26.252521 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.252497    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:26.262671 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.262641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bea06bea-ce5d-42a6-ad14-3025bee0c0a7-original-pull-secret\") pod \"global-pull-secret-syncer-bztwt\" (UID: \"bea06bea-ce5d-42a6-ad14-3025bee0c0a7\") " pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:10:26.323354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.323323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bztwt"
Jul 28 17:10:26.350991 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.350966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:10:26.353479 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.353460    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:26.363586 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.363568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c-metrics-certs\") pod \"network-metrics-daemon-zk8h7\" (UID: \"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c\") " pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:10:26.451448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.451410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:10:26.454137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.454111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jm2\" (UniqueName: \"kubernetes.io/projected/e663d17a-7350-4a83-9111-8d313abeb23f-kube-api-access-h6jm2\") pod \"network-check-target-jtqn7\" (UID: \"e663d17a-7350-4a83-9111-8d313abeb23f\") " pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:10:26.615849 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.615820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mz7f2\""
Jul 28 17:10:26.623968 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.623937    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:10:26.632396 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.632368    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nvllg\""
Jul 28 17:10:26.640787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:26.640768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zk8h7"
Jul 28 17:10:28.330024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:28.329994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bztwt"]
Jul 28 17:10:28.334314 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:28.334278    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea06bea_ce5d_42a6_ad14_3025bee0c0a7.slice/crio-57d8dd847fb0119543eb6ad2c877391759c420b55113d73840906418d6f7f3dd WatchSource:0}: Error finding container 57d8dd847fb0119543eb6ad2c877391759c420b55113d73840906418d6f7f3dd: Status 404 returned error can't find the container with id 57d8dd847fb0119543eb6ad2c877391759c420b55113d73840906418d6f7f3dd
Jul 28 17:10:28.351053 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:28.351016    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jtqn7"]
Jul 28 17:10:28.352886 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:28.352847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode663d17a_7350_4a83_9111_8d313abeb23f.slice/crio-62ec7abf3dcd57231aee6a5e402a75321ae82f617bffe78de94af0877084b711 WatchSource:0}: Error finding container 62ec7abf3dcd57231aee6a5e402a75321ae82f617bffe78de94af0877084b711: Status 404 returned error can't find the container with id 62ec7abf3dcd57231aee6a5e402a75321ae82f617bffe78de94af0877084b711
Jul 28 17:10:28.368130 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:28.368097    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zk8h7"]
Jul 28 17:10:28.370982 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:28.370955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482a185b_1b91_4c17_bb2c_ad9bf0cc1d6c.slice/crio-133ffe52ab53546c251f55ddd972c7a6eb369b205bb7f7f3c7e3a03ad44d7462 WatchSource:0}: Error finding container 133ffe52ab53546c251f55ddd972c7a6eb369b205bb7f7f3c7e3a03ad44d7462: Status 404 returned error can't find the container with id 133ffe52ab53546c251f55ddd972c7a6eb369b205bb7f7f3c7e3a03ad44d7462
Jul 28 17:10:29.077551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.077514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bztwt" event={"ID":"bea06bea-ce5d-42a6-ad14-3025bee0c0a7","Type":"ContainerStarted","Data":"57d8dd847fb0119543eb6ad2c877391759c420b55113d73840906418d6f7f3dd"}
Jul 28 17:10:29.078913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.078885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jtqn7" event={"ID":"e663d17a-7350-4a83-9111-8d313abeb23f","Type":"ContainerStarted","Data":"08b32528a4674999a9231d35d5d2742e771dde07376be7736d65e6e75e89b40e"}
Jul 28 17:10:29.079029 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.078927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jtqn7" event={"ID":"e663d17a-7350-4a83-9111-8d313abeb23f","Type":"ContainerStarted","Data":"62ec7abf3dcd57231aee6a5e402a75321ae82f617bffe78de94af0877084b711"}
Jul 28 17:10:29.079029 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.078968    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:10:29.080028 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.080009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zk8h7" event={"ID":"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c","Type":"ContainerStarted","Data":"133ffe52ab53546c251f55ddd972c7a6eb369b205bb7f7f3c7e3a03ad44d7462"}
Jul 28 17:10:29.081220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.081194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gg8k" event={"ID":"42dc4d4c-4a18-4939-99cf-85b7e1b368c3","Type":"ContainerStarted","Data":"811e6f6ae8c6fe08da5e9d41621ef54cccc46ce00bc6a780d03aa903e9354878"}
Jul 28 17:10:29.101198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.101156    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jtqn7" podStartSLOduration=67.101143825 podStartE2EDuration="1m7.101143825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:29.100194542 +0000 UTC m=+67.982601659" watchObservedRunningTime="2026-07-28 17:10:29.101143825 +0000 UTC m=+67.983550942"
Jul 28 17:10:29.379699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:29.379699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:29.380196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:29.380196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:29.380196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:29.380196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.379996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:29.380439 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.380363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d142323f-846b-471d-82b6-5dfbed8e5128-service-ca-bundle\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:29.382854 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.382830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d136840f-e10d-4846-9f16-ffbc2ae86c70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-njt9d\" (UID: \"d136840f-e10d-4846-9f16-ffbc2ae86c70\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:29.382854 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.382846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3db7855-e235-4278-9b62-79e43b6fcb11-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-97z85\" (UID: \"d3db7855-e235-4278-9b62-79e43b6fcb11\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:29.383039 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.382931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024d38b7-c1c9-4e99-847d-5d65c15530d7-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-8wdww\" (UID: \"024d38b7-c1c9-4e99-847d-5d65c15530d7\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:29.383801 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.383679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d142323f-846b-471d-82b6-5dfbed8e5128-metrics-certs\") pod \"router-default-66cf49fbb7-drwbq\" (UID: \"d142323f-846b-471d-82b6-5dfbed8e5128\") " pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:29.383801 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.383753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"image-registry-7f76759cf4-rz9l8\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") " pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:29.481447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.481401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:29.484408 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.484380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb53b3e-7d01-4584-b233-2bb144f857be-metrics-tls\") pod \"dns-default-tstvj\" (UID: \"ccb53b3e-7d01-4584-b233-2bb144f857be\") " pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:29.555285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.555252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6jkxz\""
Jul 28 17:10:29.562622 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.562599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:29.584420 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.584201    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rqz7s\""
Jul 28 17:10:29.591530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.591301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b4zkw\""
Jul 28 17:10:29.591530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.591490    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:29.599716 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.599688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"
Jul 28 17:10:29.621608 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.621558    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9bmhj\""
Jul 28 17:10:29.629295 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.629238    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"
Jul 28 17:10:29.635252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.635042    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9lttv\""
Jul 28 17:10:29.643473 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.643419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"
Jul 28 17:10:29.657169 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.656601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8842r\""
Jul 28 17:10:29.660195 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.660027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:29.807356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.807055    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:10:29.838510 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:29.838461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ced5784_9221_42e6_9012_315380e55baf.slice/crio-c357adef7fb33784064ed91053a9185cbee3bbaa5d81b5eb45743494ffa3d419 WatchSource:0}: Error finding container c357adef7fb33784064ed91053a9185cbee3bbaa5d81b5eb45743494ffa3d419: Status 404 returned error can't find the container with id c357adef7fb33784064ed91053a9185cbee3bbaa5d81b5eb45743494ffa3d419
Jul 28 17:10:29.891514 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.889237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66cf49fbb7-drwbq"]
Jul 28 17:10:29.892496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.892452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d"]
Jul 28 17:10:29.902083 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:29.902047    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd136840f_e10d_4846_9f16_ffbc2ae86c70.slice/crio-38e172dd3040a80fe3ebd66003d0a8f0b2e6f1caa724e002e16c8b11f9ac0ffa WatchSource:0}: Error finding container 38e172dd3040a80fe3ebd66003d0a8f0b2e6f1caa724e002e16c8b11f9ac0ffa: Status 404 returned error can't find the container with id 38e172dd3040a80fe3ebd66003d0a8f0b2e6f1caa724e002e16c8b11f9ac0ffa
Jul 28 17:10:29.937497 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.937466    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85"]
Jul 28 17:10:29.956196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.956117    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-8wdww"]
Jul 28 17:10:29.961365 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:29.961307    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024d38b7_c1c9_4e99_847d_5d65c15530d7.slice/crio-2eeb1f85967bd7e67dc7b8303816e099b31d36f55e7423ab7e409dac25c04faa WatchSource:0}: Error finding container 2eeb1f85967bd7e67dc7b8303816e099b31d36f55e7423ab7e409dac25c04faa: Status 404 returned error can't find the container with id 2eeb1f85967bd7e67dc7b8303816e099b31d36f55e7423ab7e409dac25c04faa
Jul 28 17:10:29.975046 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:29.974695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tstvj"]
Jul 28 17:10:29.981680 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:29.981647    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb53b3e_7d01_4584_b233_2bb144f857be.slice/crio-7679a03c92b80baa27f177ac23b98f0a8573c75c9d2c5be2948225cc951c637d WatchSource:0}: Error finding container 7679a03c92b80baa27f177ac23b98f0a8573c75c9d2c5be2948225cc951c637d: Status 404 returned error can't find the container with id 7679a03c92b80baa27f177ac23b98f0a8573c75c9d2c5be2948225cc951c637d
Jul 28 17:10:30.087450 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.087404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85" event={"ID":"d3db7855-e235-4278-9b62-79e43b6fcb11","Type":"ContainerStarted","Data":"f69c2751b2acf345f9b7fea051c9a33862374a832841737da76c6d2435906cab"}
Jul 28 17:10:30.089386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.089355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66cf49fbb7-drwbq" event={"ID":"d142323f-846b-471d-82b6-5dfbed8e5128","Type":"ContainerStarted","Data":"7adf7fd3d00f555ae8f2ddd21e69c30a3a424526c4e80192ce20d32e6a312eb6"}
Jul 28 17:10:30.089502 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.089406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66cf49fbb7-drwbq" event={"ID":"d142323f-846b-471d-82b6-5dfbed8e5128","Type":"ContainerStarted","Data":"8540ff410e9de87239c4f563def128c4d6a4fe0c372f0612169e4635c8e68a60"}
Jul 28 17:10:30.091478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.091455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" event={"ID":"4ced5784-9221-42e6-9012-315380e55baf","Type":"ContainerStarted","Data":"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"}
Jul 28 17:10:30.091578 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.091498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" event={"ID":"4ced5784-9221-42e6-9012-315380e55baf","Type":"ContainerStarted","Data":"c357adef7fb33784064ed91053a9185cbee3bbaa5d81b5eb45743494ffa3d419"}
Jul 28 17:10:30.091578 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.091556    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:30.093448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.093398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww" event={"ID":"024d38b7-c1c9-4e99-847d-5d65c15530d7","Type":"ContainerStarted","Data":"2eeb1f85967bd7e67dc7b8303816e099b31d36f55e7423ab7e409dac25c04faa"}
Jul 28 17:10:30.094777 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.094750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d" event={"ID":"d136840f-e10d-4846-9f16-ffbc2ae86c70","Type":"ContainerStarted","Data":"38e172dd3040a80fe3ebd66003d0a8f0b2e6f1caa724e002e16c8b11f9ac0ffa"}
Jul 28 17:10:30.096099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.096078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tstvj" event={"ID":"ccb53b3e-7d01-4584-b233-2bb144f857be","Type":"ContainerStarted","Data":"7679a03c92b80baa27f177ac23b98f0a8573c75c9d2c5be2948225cc951c637d"}
Jul 28 17:10:30.112543 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.112494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66cf49fbb7-drwbq" podStartSLOduration=66.11247714 podStartE2EDuration="1m6.11247714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:30.112434154 +0000 UTC m=+68.994841268" watchObservedRunningTime="2026-07-28 17:10:30.11247714 +0000 UTC m=+68.994884259"
Jul 28 17:10:30.132209 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.132168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" podStartSLOduration=68.132152948 podStartE2EDuration="1m8.132152948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:30.131177359 +0000 UTC m=+69.013584481" watchObservedRunningTime="2026-07-28 17:10:30.132152948 +0000 UTC m=+69.014560064"
Jul 28 17:10:30.592153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.592033    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:30.594859 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.594764    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:30.740779 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.740733    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-mj5xh"]
Jul 28 17:10:30.743605 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.743553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:30.746457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.746431    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:10:30.747305 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.746923    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dp6tw\""
Jul 28 17:10:30.747305 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.747150    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:10:30.755559 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.755538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mj5xh"]
Jul 28 17:10:30.896259 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:30.896228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxc4d\" (UniqueName: \"kubernetes.io/projected/78974b3b-f44f-4b4b-9957-c35bf777fea2-kube-api-access-sxc4d\") pod \"downloads-7c97dc4555-mj5xh\" (UID: \"78974b3b-f44f-4b4b-9957-c35bf777fea2\") " pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:31.000631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.000148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxc4d\" (UniqueName: \"kubernetes.io/projected/78974b3b-f44f-4b4b-9957-c35bf777fea2-kube-api-access-sxc4d\") pod \"downloads-7c97dc4555-mj5xh\" (UID: \"78974b3b-f44f-4b4b-9957-c35bf777fea2\") " pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:31.021698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.021636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxc4d\" (UniqueName: \"kubernetes.io/projected/78974b3b-f44f-4b4b-9957-c35bf777fea2-kube-api-access-sxc4d\") pod \"downloads-7c97dc4555-mj5xh\" (UID: \"78974b3b-f44f-4b4b-9957-c35bf777fea2\") " pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:31.057320 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.057195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:31.111881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.110200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zk8h7" event={"ID":"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c","Type":"ContainerStarted","Data":"d6fbfa940df7e8c0ad81557230539d3204b24ec4ad15a3c7fb2df19cb3d5c570"}
Jul 28 17:10:31.111881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.110247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zk8h7" event={"ID":"482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c","Type":"ContainerStarted","Data":"fb199a0824d6066511cc9750a9adb8f9930de7149efed8ae2a1456fc189d33ba"}
Jul 28 17:10:31.111881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.110306    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:31.112969 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.112921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66cf49fbb7-drwbq"
Jul 28 17:10:31.135644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.134165    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zk8h7" podStartSLOduration=66.755260371 podStartE2EDuration="1m9.130909456s" totalImagesPullingTime="2.375649085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:31.13017216 +0000 UTC m=+70.012579280" watchObservedRunningTime="2026-07-28 17:10:31.130909456 +0000 UTC m=+70.013316574"
Jul 28 17:10:31.270360 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:31.270326    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mj5xh"]
Jul 28 17:10:31.284258 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:31.278286    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78974b3b_f44f_4b4b_9957_c35bf777fea2.slice/crio-295593ce3cca7ae8c806663463e53c7666d39f7694f0f5b75a625ff9c1d1ceb7 WatchSource:0}: Error finding container 295593ce3cca7ae8c806663463e53c7666d39f7694f0f5b75a625ff9c1d1ceb7: Status 404 returned error can't find the container with id 295593ce3cca7ae8c806663463e53c7666d39f7694f0f5b75a625ff9c1d1ceb7
Jul 28 17:10:32.117863 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:32.117528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mj5xh" event={"ID":"78974b3b-f44f-4b4b-9957-c35bf777fea2","Type":"ContainerStarted","Data":"295593ce3cca7ae8c806663463e53c7666d39f7694f0f5b75a625ff9c1d1ceb7"}
Jul 28 17:10:39.146313 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.146259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww" event={"ID":"024d38b7-c1c9-4e99-847d-5d65c15530d7","Type":"ContainerStarted","Data":"851ca1e9720ce726de3406de4c36abe13bc5282f5a9accdf724d9b6c0da60e74"}
Jul 28 17:10:39.148180 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.148155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d" event={"ID":"d136840f-e10d-4846-9f16-ffbc2ae86c70","Type":"ContainerStarted","Data":"51d4f367e1fc1a4c9b45a8c027f72363f984474441c067179eaf8e51b23e6f03"}
Jul 28 17:10:39.150191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.150167    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gg8k" event={"ID":"42dc4d4c-4a18-4939-99cf-85b7e1b368c3","Type":"ContainerStarted","Data":"84006731a7a5daf181529c954cf28650fe17f516984e39fa0a371ae579b50131"}
Jul 28 17:10:39.152429 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.152238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tstvj" event={"ID":"ccb53b3e-7d01-4584-b233-2bb144f857be","Type":"ContainerStarted","Data":"a156199af191291ce6de565b232d95c6622b83658525926276c7200404943a32"}
Jul 28 17:10:39.152429 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.152291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:39.152429 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.152316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tstvj" event={"ID":"ccb53b3e-7d01-4584-b233-2bb144f857be","Type":"ContainerStarted","Data":"ac9f6a57ab5f7c74238cd8bf433fcc3b3c4dca6a2daac97da35fb07ec933d853"}
Jul 28 17:10:39.153994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.153970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bztwt" event={"ID":"bea06bea-ce5d-42a6-ad14-3025bee0c0a7","Type":"ContainerStarted","Data":"cd35f97bc7ac5703bcfd9c2437a0a51ffd3af56cb594983079223c8536d89284"}
Jul 28 17:10:39.155825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.155804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85" event={"ID":"d3db7855-e235-4278-9b62-79e43b6fcb11","Type":"ContainerStarted","Data":"91c8a0e9e17a6fccdbc778c93855c61c9554179e68874536c74fee7dd10eb91e"}
Jul 28 17:10:39.155942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.155845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85" event={"ID":"d3db7855-e235-4278-9b62-79e43b6fcb11","Type":"ContainerStarted","Data":"33190d8f1e4d262b6ddd6a3dd885f430b9cbeee2f2747d9beeb24acb53c5e8b2"}
Jul 28 17:10:39.164656 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.164611    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-8wdww" podStartSLOduration=59.771259426 podStartE2EDuration="1m8.164593339s" totalImagesPullingTime="8.393333913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.162604139 +0000 UTC m=+78.045011256" watchObservedRunningTime="2026-07-28 17:10:39.164593339 +0000 UTC m=+78.047000434"
Jul 28 17:10:39.184299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.184235    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tstvj" podStartSLOduration=33.808093508 podStartE2EDuration="42.184198854s" totalImagesPullingTime="8.376105346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.182603075 +0000 UTC m=+78.065010189" watchObservedRunningTime="2026-07-28 17:10:39.184198854 +0000 UTC m=+78.066605970"
Jul 28 17:10:39.203145 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.202753    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-97z85" podStartSLOduration=66.864639327 podStartE2EDuration="1m15.202732512s" totalImagesPullingTime="8.338093185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.202368574 +0000 UTC m=+78.084775694" watchObservedRunningTime="2026-07-28 17:10:39.202732512 +0000 UTC m=+78.085139630"
Jul 28 17:10:39.221241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:39.219865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bztwt" podStartSLOduration=67.19780088 podStartE2EDuration="1m17.219847076s" totalImagesPullingTime="10.022046196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.218772319 +0000 UTC m=+78.101179445" watchObservedRunningTime="2026-07-28 17:10:39.219847076 +0000 UTC m=+78.102254193"
Jul 28 17:10:43.176247 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:43.176156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gg8k" event={"ID":"42dc4d4c-4a18-4939-99cf-85b7e1b368c3","Type":"ContainerStarted","Data":"4a8e819322cf67df91087559e7846122c8b25fe261ca4fc45324b451d1b8557e"}
Jul 28 17:10:43.202727 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:43.202681    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5gg8k" podStartSLOduration=16.995542339 podStartE2EDuration="36.202665406s" totalImagesPullingTime="19.207123067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:43.201613573 +0000 UTC m=+82.084020748" watchObservedRunningTime="2026-07-28 17:10:43.202665406 +0000 UTC m=+82.085072521"
Jul 28 17:10:43.203858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:43.203818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-njt9d" podStartSLOduration=70.747164644 podStartE2EDuration="1m19.203808407s" totalImagesPullingTime="8.456643763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.241203548 +0000 UTC m=+78.123610663" watchObservedRunningTime="2026-07-28 17:10:43.203808407 +0000 UTC m=+82.086215583"
Jul 28 17:10:49.165025 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:49.164991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tstvj"
Jul 28 17:10:50.537360 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.537326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"]
Jul 28 17:10:50.539889 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.539857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.543330 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.543311    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:10:50.544016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.543997    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:10:50.545038 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.545018    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r9thq\""
Jul 28 17:10:50.545038 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.545036    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:10:50.555364 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.555342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"]
Jul 28 17:10:50.572860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.572830    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vbk6s"]
Jul 28 17:10:50.575537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.575519    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.578865 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.578841    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:10:50.579164 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.579147    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mdzx4\""
Jul 28 17:10:50.579374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.579358    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:10:50.579481 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.579465    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:10:50.649636 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-root\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.649811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-sys\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.649811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.649811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-wtmp\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.649811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650021 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-textfile\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650021 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650021 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.650021 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.649970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.650021 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.650010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-metrics-client-ca\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.650035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdfb\" (UniqueName: \"kubernetes.io/projected/f79b973c-4627-4f33-bde7-7d7f29a4be69-kube-api-access-dbdfb\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.650097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.650216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.650124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44rh\" (UniqueName: \"kubernetes.io/projected/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-kube-api-access-z44rh\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.750772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.750772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-metrics-client-ca\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdfb\" (UniqueName: \"kubernetes.io/projected/f79b973c-4627-4f33-bde7-7d7f29a4be69-kube-api-access-dbdfb\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44rh\" (UniqueName: \"kubernetes.io/projected/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-kube-api-access-z44rh\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.750990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-root\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-sys\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751392 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.751392 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-wtmp\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751392 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751392 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-textfile\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751588 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.751523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-textfile\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.751697 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:50.751649    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:10:50.751749 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:10:50.751715    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls podName:f79b973c-4627-4f33-bde7-7d7f29a4be69 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.251693097 +0000 UTC m=+90.134100198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls") pod "node-exporter-vbk6s" (UID: "f79b973c-4627-4f33-bde7-7d7f29a4be69") : secret "node-exporter-tls" not found
Jul 28 17:10:50.752424 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.752382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-root\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.753433 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.752828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-wtmp\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.753433 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.752905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f79b973c-4627-4f33-bde7-7d7f29a4be69-sys\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.753611 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.753555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.754119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.754097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.755004 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.754742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f79b973c-4627-4f33-bde7-7d7f29a4be69-metrics-client-ca\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.755720 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.755696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.757676 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.757634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.760935 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.760915    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.764823 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.764756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdfb\" (UniqueName: \"kubernetes.io/projected/f79b973c-4627-4f33-bde7-7d7f29a4be69-kube-api-access-dbdfb\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:50.766829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.766808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44rh\" (UniqueName: \"kubernetes.io/projected/c33669e0-c5b9-492c-8f7b-d4bbb74a9e00-kube-api-access-z44rh\") pod \"openshift-state-metrics-b8f49f5c6-m8zfc\" (UID: \"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:50.849407 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:50.849375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"
Jul 28 17:10:51.021245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.021166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc"]
Jul 28 17:10:51.025475 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:51.025444    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33669e0_c5b9_492c_8f7b_d4bbb74a9e00.slice/crio-a7fa976b76e75178e1914a9950bb247eddd0ce6f3a9fb0a5c5ce382498ca3b80 WatchSource:0}: Error finding container a7fa976b76e75178e1914a9950bb247eddd0ce6f3a9fb0a5c5ce382498ca3b80: Status 404 returned error can't find the container with id a7fa976b76e75178e1914a9950bb247eddd0ce6f3a9fb0a5c5ce382498ca3b80
Jul 28 17:10:51.117814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.117529    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:10:51.221764 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.221725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc" event={"ID":"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00","Type":"ContainerStarted","Data":"c7e52c3b26fca7c1f4ce7b526f1e4009043f051cef523820b965c34fae4af868"}
Jul 28 17:10:51.221903 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.221802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc" event={"ID":"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00","Type":"ContainerStarted","Data":"4a8a8bb4b427e14c7c3c1fab9df7f87eb3710fe705d81363874f73313a267264"}
Jul 28 17:10:51.221903 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.221836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc" event={"ID":"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00","Type":"ContainerStarted","Data":"a7fa976b76e75178e1914a9950bb247eddd0ce6f3a9fb0a5c5ce382498ca3b80"}
Jul 28 17:10:51.223770 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.223741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mj5xh" event={"ID":"78974b3b-f44f-4b4b-9957-c35bf777fea2","Type":"ContainerStarted","Data":"2bbb20a21a2af47471d92ab646699ed16acc31ec4577a70663bfc510e17a5172"}
Jul 28 17:10:51.225065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.224746    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:51.238947 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.238926    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-mj5xh"
Jul 28 17:10:51.246815 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.246766    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-mj5xh" podStartSLOduration=2.07670344 podStartE2EDuration="21.246750106s" totalImagesPullingTime="19.170046666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:51.24465541 +0000 UTC m=+90.127062525" watchObservedRunningTime="2026-07-28 17:10:51.246750106 +0000 UTC m=+90.129157221"
Jul 28 17:10:51.256290 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.256265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:51.258941 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.258919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f79b973c-4627-4f33-bde7-7d7f29a4be69-node-exporter-tls\") pod \"node-exporter-vbk6s\" (UID: \"f79b973c-4627-4f33-bde7-7d7f29a4be69\") " pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:51.484755 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:51.484720    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbk6s"
Jul 28 17:10:51.512533 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:51.512492    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79b973c_4627_4f33_bde7_7d7f29a4be69.slice/crio-609e3dc2f9224aa6b7108f343b533f27cadbcf4ff77842e50ccc350b4d1e7dc0 WatchSource:0}: Error finding container 609e3dc2f9224aa6b7108f343b533f27cadbcf4ff77842e50ccc350b4d1e7dc0: Status 404 returned error can't find the container with id 609e3dc2f9224aa6b7108f343b533f27cadbcf4ff77842e50ccc350b4d1e7dc0
Jul 28 17:10:52.232164 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:52.232135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbk6s" event={"ID":"f79b973c-4627-4f33-bde7-7d7f29a4be69","Type":"ContainerStarted","Data":"609e3dc2f9224aa6b7108f343b533f27cadbcf4ff77842e50ccc350b4d1e7dc0"}
Jul 28 17:10:53.793606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:53.793573    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:10:54.248427 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.248134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc" event={"ID":"c33669e0-c5b9-492c-8f7b-d4bbb74a9e00","Type":"ContainerStarted","Data":"53e958b82c1fc4b7af7edace0ce52fca56038bf4248c46e232b00a4a26fedb9b"}
Jul 28 17:10:54.250068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.250040    2559 generic.go:356] "Generic (PLEG): container finished" podID="f79b973c-4627-4f33-bde7-7d7f29a4be69" containerID="1e7c29d53d3b5b63ac23fffe2962ab58c818efb8d278131f2de442e00c8ac816" exitCode=0
Jul 28 17:10:54.250214 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.250103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbk6s" event={"ID":"f79b973c-4627-4f33-bde7-7d7f29a4be69","Type":"ContainerDied","Data":"1e7c29d53d3b5b63ac23fffe2962ab58c818efb8d278131f2de442e00c8ac816"}
Jul 28 17:10:54.268249 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.268203    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m8zfc" podStartSLOduration=1.838140377 podStartE2EDuration="4.268186482s" totalImagesPullingTime="2.430046105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:54.265680726 +0000 UTC m=+93.148087841" watchObservedRunningTime="2026-07-28 17:10:54.268186482 +0000 UTC m=+93.150593598"
Jul 28 17:10:54.645218 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.645179    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-999494bd7-hgj8z"]
Jul 28 17:10:54.668463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.668435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-999494bd7-hgj8z"]
Jul 28 17:10:54.668635 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.668550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.671185 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671147    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 17:10:54.671312 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 17:10:54.671312 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 17:10:54.671429 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671316    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 17:10:54.671561 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671540    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 17:10:54.671688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bhjn8\""
Jul 28 17:10:54.671688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.671601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a9rtferkme5ij\""
Jul 28 17:10:54.789016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.788926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.788985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a375a6fc-eaeb-4832-a51a-0277bc208f39-metrics-client-ca\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-grpc-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5r7\" (UniqueName: \"kubernetes.io/projected/a375a6fc-eaeb-4832-a51a-0277bc208f39-kube-api-access-ww5r7\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.789325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.789271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.864781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.864751    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65975f975f-n4kv6"]
Jul 28 17:10:54.881009 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.880981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65975f975f-n4kv6"]
Jul 28 17:10:54.881154 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.881122    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.885842 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.885703    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:10:54.886006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.885898    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2c7x7\""
Jul 28 17:10:54.886175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.886156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:10:54.886311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.886292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:10:54.886543 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.886522    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8pb0qlgntbfe4\""
Jul 28 17:10:54.886642 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.886577    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:10:54.889918 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.889918 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.889918 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.889918 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a375a6fc-eaeb-4832-a51a-0277bc208f39-metrics-client-ca\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.889918 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-grpc-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.890223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5r7\" (UniqueName: \"kubernetes.io/projected/a375a6fc-eaeb-4832-a51a-0277bc208f39-kube-api-access-ww5r7\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.890223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.889969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.890223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.890000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.892310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.892285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a375a6fc-eaeb-4832-a51a-0277bc208f39-metrics-client-ca\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.897914 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.897735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.898271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.898251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.899141 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.898911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-grpc-tls\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.899141 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.899102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.899586 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.899567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.900233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.900216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a375a6fc-eaeb-4832-a51a-0277bc208f39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.901400 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.901380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5r7\" (UniqueName: \"kubernetes.io/projected/a375a6fc-eaeb-4832-a51a-0277bc208f39-kube-api-access-ww5r7\") pod \"thanos-querier-999494bd7-hgj8z\" (UID: \"a375a6fc-eaeb-4832-a51a-0277bc208f39\") " pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.980549 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.980516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:10:54.990853 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.990817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991003 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.990923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-client-certs\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991003 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.990960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-audit-log\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991003 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.990993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-tls\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.991016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-metrics-server-audit-profiles\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.991037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-client-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:54.991147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:54.991094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-kube-api-access-smt22\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-audit-log\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-tls\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-metrics-server-audit-profiles\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-client-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-kube-api-access-smt22\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.092973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.092511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-client-certs\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.095210 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.094450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-metrics-server-audit-profiles\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.095210 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.094755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-audit-log\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.095964 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.095630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.096631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.096615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-client-certs\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.099427 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.099035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-client-ca-bundle\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.099427 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.099389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-secret-metrics-server-tls\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.104896 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.104828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt22\" (UniqueName: \"kubernetes.io/projected/7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f-kube-api-access-smt22\") pod \"metrics-server-65975f975f-n4kv6\" (UID: \"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f\") " pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.187643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.187601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-999494bd7-hgj8z"]
Jul 28 17:10:55.195035 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.195009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:10:55.260288 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.259349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbk6s" event={"ID":"f79b973c-4627-4f33-bde7-7d7f29a4be69","Type":"ContainerStarted","Data":"8852dd36838112971b0889fb56f1deb2eaa61848a5a479950b1cb6d48df378b0"}
Jul 28 17:10:55.260288 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.259413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbk6s" event={"ID":"f79b973c-4627-4f33-bde7-7d7f29a4be69","Type":"ContainerStarted","Data":"14448b958c89e52c44afeb4759cd0bfa032d7f989e70042dacb995d5393633a2"}
Jul 28 17:10:55.272474 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.271909    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"fc737a61ad8303bc9c38bffeba0f3fa4ecd33b940cfc333dee9b21428f44fa03"}
Jul 28 17:10:55.315308 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.315258    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vbk6s" podStartSLOduration=3.090707038 podStartE2EDuration="5.315234412s" totalImagesPullingTime="2.123960524s" totalInitContainerRuntime="100.56685ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:55.287136217 +0000 UTC m=+94.169543335" watchObservedRunningTime="2026-07-28 17:10:55.315234412 +0000 UTC m=+94.197641529"
Jul 28 17:10:55.315643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.315622    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"]
Jul 28 17:10:55.343126 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.343093    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:10:55.343436 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.343408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.353410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.353369    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-njxxg\""
Jul 28 17:10:55.353963 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.353909    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:10:55.369025 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.368974    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"]
Jul 28 17:10:55.369025 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.369014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:10:55.369202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.369108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.372755 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.372235    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fcpnl\""
Jul 28 17:10:55.372755 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.372535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:10:55.373031 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.372804    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:10:55.373031 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.373052    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:10:55.373390 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.373356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:10:55.373594 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.373563    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:10:55.379131 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.379113    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:10:55.394929 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.394845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.394929 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.394916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.395084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.394942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkhc\" (UniqueName: \"kubernetes.io/projected/6a3b3e13-869e-4920-b052-f50f8a23f11b-kube-api-access-jjkhc\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.395084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.394973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.395084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.394989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a3b3e13-869e-4920-b052-f50f8a23f11b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.395084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.395057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.395246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.395120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.395246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.395147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.395246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.395199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rgw\" (UniqueName: \"kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.405846 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.405820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65975f975f-n4kv6"]
Jul 28 17:10:55.408392 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:55.408368    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cffa9e1_77da_4d2d_b09b_4f6c82d6fc6f.slice/crio-e8dc98f09e7c8378d6232dc03dda04380a25922d59834709705e9d77c9eb0635 WatchSource:0}: Error finding container e8dc98f09e7c8378d6232dc03dda04380a25922d59834709705e9d77c9eb0635: Status 404 returned error can't find the container with id e8dc98f09e7c8378d6232dc03dda04380a25922d59834709705e9d77c9eb0635
Jul 28 17:10:55.496525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rgw\" (UniqueName: \"kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkhc\" (UniqueName: \"kubernetes.io/projected/6a3b3e13-869e-4920-b052-f50f8a23f11b-kube-api-access-jjkhc\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.496697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a3b3e13-869e-4920-b052-f50f8a23f11b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.496980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.496980 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.496777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.497600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.497421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.497600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.497429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.497794 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.497746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.498066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.498021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.499779 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.499754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.499890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.499755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a3b3e13-869e-4920-b052-f50f8a23f11b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.499960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.499886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.506271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.506252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkhc\" (UniqueName: \"kubernetes.io/projected/6a3b3e13-869e-4920-b052-f50f8a23f11b-kube-api-access-jjkhc\") pod \"monitoring-plugin-68b7fc9574-hwn2z\" (UID: \"6a3b3e13-869e-4920-b052-f50f8a23f11b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.506441 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.506423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rgw\" (UniqueName: \"kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw\") pod \"console-5f9b477fd4-l56m4\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") " pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.660612 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.660541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:10:55.682857 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.682835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:10:55.865731 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.865590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"]
Jul 28 17:10:55.869493 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:55.869450    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3b3e13_869e_4920_b052_f50f8a23f11b.slice/crio-8fcd2669fab7ab263532a2d16fe326cef4b5bb9a97b3e24ba348bf2d83936d2a WatchSource:0}: Error finding container 8fcd2669fab7ab263532a2d16fe326cef4b5bb9a97b3e24ba348bf2d83936d2a: Status 404 returned error can't find the container with id 8fcd2669fab7ab263532a2d16fe326cef4b5bb9a97b3e24ba348bf2d83936d2a
Jul 28 17:10:55.899250 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:55.899226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:10:55.900383 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:10:55.900357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9621deaa_88dd_4cd9_a82b_701ac688602e.slice/crio-be95e4a77bf64ace051e36be565250c7fe569bfcb589a4ae26074e458a83c315 WatchSource:0}: Error finding container be95e4a77bf64ace051e36be565250c7fe569bfcb589a4ae26074e458a83c315: Status 404 returned error can't find the container with id be95e4a77bf64ace051e36be565250c7fe569bfcb589a4ae26074e458a83c315
Jul 28 17:10:56.280431 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:56.280202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9b477fd4-l56m4" event={"ID":"9621deaa-88dd-4cd9-a82b-701ac688602e","Type":"ContainerStarted","Data":"be95e4a77bf64ace051e36be565250c7fe569bfcb589a4ae26074e458a83c315"}
Jul 28 17:10:56.282522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:56.282492    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z" event={"ID":"6a3b3e13-869e-4920-b052-f50f8a23f11b","Type":"ContainerStarted","Data":"8fcd2669fab7ab263532a2d16fe326cef4b5bb9a97b3e24ba348bf2d83936d2a"}
Jul 28 17:10:56.284323 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:10:56.284256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6" event={"ID":"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f","Type":"ContainerStarted","Data":"e8dc98f09e7c8378d6232dc03dda04380a25922d59834709705e9d77c9eb0635"}
Jul 28 17:11:00.099614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:00.099580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jtqn7"
Jul 28 17:11:01.324270 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.324207    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9b477fd4-l56m4" event={"ID":"9621deaa-88dd-4cd9-a82b-701ac688602e","Type":"ContainerStarted","Data":"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"}
Jul 28 17:11:01.325930 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.325903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z" event={"ID":"6a3b3e13-869e-4920-b052-f50f8a23f11b","Type":"ContainerStarted","Data":"c7f7d84c07739d477c2a0ae676ab8ac1ebf119928d1d52c3e6e9efbefd87f3e0"}
Jul 28 17:11:01.326102 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.326076    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:11:01.327644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.327546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6" event={"ID":"7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f","Type":"ContainerStarted","Data":"7d9ce2f929a6fb1ddfb49eaacb3b2844ce35298ea853c6a2fe9bd2916e760be4"}
Jul 28 17:11:01.329927 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.329908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"0db61fbfee8e3d53cfde850d3a6860b4276ebb78f2e483e3f48c895579676358"}
Jul 28 17:11:01.330044 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.329957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"9ff62842447c7dcf8e9776a76306d1b80a3a3b52a2be7fe080548ab23bf210bc"}
Jul 28 17:11:01.330044 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.329990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"839dbf5d1e5e526527f9f1ae4251aab23037b1122adac0d89a42bccf748cd77d"}
Jul 28 17:11:01.331531 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.331509    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z"
Jul 28 17:11:01.345709 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.345655    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f9b477fd4-l56m4" podStartSLOduration=1.190772854 podStartE2EDuration="6.345642202s" totalImagesPullingTime="5.154869348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:01.344249268 +0000 UTC m=+100.226656386" watchObservedRunningTime="2026-07-28 17:11:01.345642202 +0000 UTC m=+100.228049318"
Jul 28 17:11:01.363714 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.363668    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-hwn2z" podStartSLOduration=1.531765342 podStartE2EDuration="6.363654016s" totalImagesPullingTime="4.831888674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:01.362120522 +0000 UTC m=+100.244527631" watchObservedRunningTime="2026-07-28 17:11:01.363654016 +0000 UTC m=+100.246061110"
Jul 28 17:11:01.386388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:01.386123    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6" podStartSLOduration=2.094437338 podStartE2EDuration="7.386104408s" totalImagesPullingTime="5.29166707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:01.385461323 +0000 UTC m=+100.267868440" watchObservedRunningTime="2026-07-28 17:11:01.386104408 +0000 UTC m=+100.268511518"
Jul 28 17:11:03.345956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:03.345921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"a3c1f2d1f03997198c681f09c42a41894dccf5bbfceedea7b3ab4c51eb7e85d7"}
Jul 28 17:11:03.346497 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:03.346010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"0526b889995ca110f7729c9e6ebb4062858dff9e58f69384672f060f6854f0ea"}
Jul 28 17:11:03.346497 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:03.346036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" event={"ID":"a375a6fc-eaeb-4832-a51a-0277bc208f39","Type":"ContainerStarted","Data":"05637584ca5225b1b6ae99cdcc10f063b6afff48cd4132a469afa118f9de1470"}
Jul 28 17:11:03.369962 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:03.369893    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z" podStartSLOduration=2.482962077 podStartE2EDuration="9.369864356s" totalImagesPullingTime="6.886902279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:03.368808194 +0000 UTC m=+102.251215320" watchObservedRunningTime="2026-07-28 17:11:03.369864356 +0000 UTC m=+102.252271472"
Jul 28 17:11:04.349949 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:04.349915    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:11:05.683660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:05.683626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:11:05.684055 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:05.683680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:11:05.688465 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:05.688446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:11:06.360724 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:06.360689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:11:10.360248 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:10.360217    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-999494bd7-hgj8z"
Jul 28 17:11:15.195188 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:15.195144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:11:15.195591 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:15.195214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:11:17.402257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.402220    2559 generic.go:356] "Generic (PLEG): container finished" podID="76dfb982-e9ff-4d68-9bcd-339a79c054db" containerID="fe88da23570a543552f28a177bc9db6dab611b6cf802848316c1d20a3f658958" exitCode=0
Jul 28 17:11:17.402844 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.402294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j" event={"ID":"76dfb982-e9ff-4d68-9bcd-339a79c054db","Type":"ContainerDied","Data":"fe88da23570a543552f28a177bc9db6dab611b6cf802848316c1d20a3f658958"}
Jul 28 17:11:17.402844 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.402649    2559 scope.go:117] "RemoveContainer" containerID="fe88da23570a543552f28a177bc9db6dab611b6cf802848316c1d20a3f658958"
Jul 28 17:11:17.403960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.403941    2559 generic.go:356] "Generic (PLEG): container finished" podID="17e5be93-25fe-40b8-a784-ba4e359a1f50" containerID="8f533f215f716728433df7f764fedf79dc72cfe6c7619bcb73ae78db1c8c6410" exitCode=0
Jul 28 17:11:17.404047 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.403994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-sr6tx" event={"ID":"17e5be93-25fe-40b8-a784-ba4e359a1f50","Type":"ContainerDied","Data":"8f533f215f716728433df7f764fedf79dc72cfe6c7619bcb73ae78db1c8c6410"}
Jul 28 17:11:17.404256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:17.404244    2559 scope.go:117] "RemoveContainer" containerID="8f533f215f716728433df7f764fedf79dc72cfe6c7619bcb73ae78db1c8c6410"
Jul 28 17:11:18.135317 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:18.135284    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66cf49fbb7-drwbq_d142323f-846b-471d-82b6-5dfbed8e5128/router/0.log"
Jul 28 17:11:18.151036 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:18.151012    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n2zb8_ddadacb7-1b96-4c8b-8e13-9f2043680423/serve-healthcheck-canary/0.log"
Jul 28 17:11:18.410361 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:18.410274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-nmw2j" event={"ID":"76dfb982-e9ff-4d68-9bcd-339a79c054db","Type":"ContainerStarted","Data":"6355561fbd626a017956641d70e728426e325c9d5f7bcf9a33eca3dc4d7d80f0"}
Jul 28 17:11:18.412146 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:18.412119    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-sr6tx" event={"ID":"17e5be93-25fe-40b8-a784-ba4e359a1f50","Type":"ContainerStarted","Data":"24ac7cc99e6e1e6dc8c7ec6486614d53c6585d79064ddca7318585d84c31b3cb"}
Jul 28 17:11:18.816841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:18.816799    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" podUID="4ced5784-9221-42e6-9012-315380e55baf" containerName="registry" containerID="cri-o://6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70" gracePeriod=30
Jul 28 17:11:19.089496 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.089434    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:11:19.214956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.214922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6mgl\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.214971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.214999    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215029    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215456 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215201    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token\") pod \"4ced5784-9221-42e6-9012-315380e55baf\" (UID: \"4ced5784-9221-42e6-9012-315380e55baf\") "
Jul 28 17:11:19.215600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215553    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:11:19.215600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.215561    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:11:19.217944 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.217911    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:11:19.218065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.217971    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:11:19.218124 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.218059    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:11:19.218183 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.218150    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:11:19.218238 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.218179    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl" (OuterVolumeSpecName: "kube-api-access-q6mgl") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "kube-api-access-q6mgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:11:19.224417 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.224393    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4ced5784-9221-42e6-9012-315380e55baf" (UID: "4ced5784-9221-42e6-9012-315380e55baf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:11:19.316134 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316102    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-bound-sa-token\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316134 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316128    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6mgl\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-kube-api-access-q6mgl\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316134 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316138    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4ced5784-9221-42e6-9012-315380e55baf-ca-trust-extracted\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316149    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4ced5784-9221-42e6-9012-315380e55baf-registry-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316158    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-image-registry-private-configuration\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316168    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-trusted-ca\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316176    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4ced5784-9221-42e6-9012-315380e55baf-installation-pull-secrets\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.316340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.316184    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4ced5784-9221-42e6-9012-315380e55baf-registry-certificates\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:11:19.417197 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.417117    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ced5784-9221-42e6-9012-315380e55baf" containerID="6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70" exitCode=0
Jul 28 17:11:19.417197 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.417164    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8"
Jul 28 17:11:19.417657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.417163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" event={"ID":"4ced5784-9221-42e6-9012-315380e55baf","Type":"ContainerDied","Data":"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"}
Jul 28 17:11:19.417657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.417288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f76759cf4-rz9l8" event={"ID":"4ced5784-9221-42e6-9012-315380e55baf","Type":"ContainerDied","Data":"c357adef7fb33784064ed91053a9185cbee3bbaa5d81b5eb45743494ffa3d419"}
Jul 28 17:11:19.417657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.417315    2559 scope.go:117] "RemoveContainer" containerID="6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"
Jul 28 17:11:19.426560 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.426544    2559 scope.go:117] "RemoveContainer" containerID="6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"
Jul 28 17:11:19.426813 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:11:19.426794    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70\": container with ID starting with 6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70 not found: ID does not exist" containerID="6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"
Jul 28 17:11:19.426905 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.426826    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70"} err="failed to get container status \"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70\": rpc error: code = NotFound desc = could not find container \"6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70\": container with ID starting with 6962c46f099aea25e376b1594b5aa50e122515f1ada082c4bd4da40a9d1c3a70 not found: ID does not exist"
Jul 28 17:11:19.442276 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.442253    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:11:19.447516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.447497    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f76759cf4-rz9l8"]
Jul 28 17:11:19.672060 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:19.671991    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ced5784-9221-42e6-9012-315380e55baf" path="/var/lib/kubelet/pods/4ced5784-9221-42e6-9012-315380e55baf/volumes"
Jul 28 17:11:22.431660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:22.431625    2559 generic.go:356] "Generic (PLEG): container finished" podID="6c594f79-fcb8-4590-bf2f-b4991d614eb8" containerID="2e3b4731a8708299fb1d5bb19119f1dd96302eea663628cb6a2f0b1fda28f3b9" exitCode=0
Jul 28 17:11:22.432087 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:22.431702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k" event={"ID":"6c594f79-fcb8-4590-bf2f-b4991d614eb8","Type":"ContainerDied","Data":"2e3b4731a8708299fb1d5bb19119f1dd96302eea663628cb6a2f0b1fda28f3b9"}
Jul 28 17:11:22.432140 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:22.432127    2559 scope.go:117] "RemoveContainer" containerID="2e3b4731a8708299fb1d5bb19119f1dd96302eea663628cb6a2f0b1fda28f3b9"
Jul 28 17:11:23.437587 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:23.437558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k" event={"ID":"6c594f79-fcb8-4590-bf2f-b4991d614eb8","Type":"ContainerStarted","Data":"05afb2c4d0ef568f6b493677951afd343e708a3103895cfc923c72653ba87c5d"}
Jul 28 17:11:35.201388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:35.201358    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:11:35.205255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:11:35.205233    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65975f975f-n4kv6"
Jul 28 17:12:27.202799 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.202725    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:12:27.203233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.203054    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ced5784-9221-42e6-9012-315380e55baf" containerName="registry"
Jul 28 17:12:27.203233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.203065    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ced5784-9221-42e6-9012-315380e55baf" containerName="registry"
Jul 28 17:12:27.203233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.203130    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ced5784-9221-42e6-9012-315380e55baf" containerName="registry"
Jul 28 17:12:27.205154 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.205137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.216723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.216698    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:12:27.285107 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.285285 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.285261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6v8q\" (UniqueName: \"kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386597 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386597 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6v8q\" (UniqueName: \"kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.386858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.386810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.387439 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.387408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.387573 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.387463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.387573 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.387469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.387676 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.387592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.389208 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.389177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.389293 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.389228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.395651 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.395630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6v8q\" (UniqueName: \"kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q\") pod \"console-84bd7f59f4-ngq8n\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") " pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.513962 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.513864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:27.645537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.645512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:12:27.648200 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:12:27.648171    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2aab90_70f8_4956_8fe0_5c5f2d89937d.slice/crio-d15707e0a5152d1b64a9986a991cdcefb0751cfaa6a5b832cd415a5b61ae5d38 WatchSource:0}: Error finding container d15707e0a5152d1b64a9986a991cdcefb0751cfaa6a5b832cd415a5b61ae5d38: Status 404 returned error can't find the container with id d15707e0a5152d1b64a9986a991cdcefb0751cfaa6a5b832cd415a5b61ae5d38
Jul 28 17:12:27.700297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:27.700267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd7f59f4-ngq8n" event={"ID":"eb2aab90-70f8-4956-8fe0-5c5f2d89937d","Type":"ContainerStarted","Data":"d15707e0a5152d1b64a9986a991cdcefb0751cfaa6a5b832cd415a5b61ae5d38"}
Jul 28 17:12:28.706253 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:28.706223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd7f59f4-ngq8n" event={"ID":"eb2aab90-70f8-4956-8fe0-5c5f2d89937d","Type":"ContainerStarted","Data":"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"}
Jul 28 17:12:28.726166 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:28.726099    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84bd7f59f4-ngq8n" podStartSLOduration=1.726084802 podStartE2EDuration="1.726084802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:28.725117063 +0000 UTC m=+187.607524180" watchObservedRunningTime="2026-07-28 17:12:28.726084802 +0000 UTC m=+187.608491917"
Jul 28 17:12:37.514906 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:37.514851    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:37.515334 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:37.514953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:37.519702 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:37.519679    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:37.749916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:37.749891    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:12:37.796518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:12:37.796452    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:13:02.818808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:02.818747    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f9b477fd4-l56m4" podUID="9621deaa-88dd-4cd9-a82b-701ac688602e" containerName="console" containerID="cri-o://da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9" gracePeriod=15
Jul 28 17:13:03.057133 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.057108    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:13:03.095027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.094948    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.094986    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5rgw\" (UniqueName: \"kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095054    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095131    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert\") pod \"9621deaa-88dd-4cd9-a82b-701ac688602e\" (UID: \"9621deaa-88dd-4cd9-a82b-701ac688602e\") "
Jul 28 17:13:03.095447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca" (OuterVolumeSpecName: "service-ca") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:03.095668 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:03.095784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095662    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:03.095784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.095724    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config" (OuterVolumeSpecName: "console-config") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:03.097573 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.097549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:03.097749 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.097725    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:03.097817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.097772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw" (OuterVolumeSpecName: "kube-api-access-z5rgw") pod "9621deaa-88dd-4cd9-a82b-701ac688602e" (UID: "9621deaa-88dd-4cd9-a82b-701ac688602e"). InnerVolumeSpecName "kube-api-access-z5rgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:03.196271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196239    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-oauth-serving-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196265    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-service-ca\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196274    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rgw\" (UniqueName: \"kubernetes.io/projected/9621deaa-88dd-4cd9-a82b-701ac688602e-kube-api-access-z5rgw\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196487 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196284    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-trusted-ca-bundle\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196487 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196292    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-serving-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196487 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196300    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9621deaa-88dd-4cd9-a82b-701ac688602e-console-oauth-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.196487 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.196308    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9621deaa-88dd-4cd9-a82b-701ac688602e-console-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:13:03.853049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.853019    2559 generic.go:356] "Generic (PLEG): container finished" podID="9621deaa-88dd-4cd9-a82b-701ac688602e" containerID="da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9" exitCode=0
Jul 28 17:13:03.853049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.853061    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f9b477fd4-l56m4"
Jul 28 17:13:03.853551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.853068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9b477fd4-l56m4" event={"ID":"9621deaa-88dd-4cd9-a82b-701ac688602e","Type":"ContainerDied","Data":"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"}
Jul 28 17:13:03.853551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.853117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f9b477fd4-l56m4" event={"ID":"9621deaa-88dd-4cd9-a82b-701ac688602e","Type":"ContainerDied","Data":"be95e4a77bf64ace051e36be565250c7fe569bfcb589a4ae26074e458a83c315"}
Jul 28 17:13:03.853551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.853143    2559 scope.go:117] "RemoveContainer" containerID="da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"
Jul 28 17:13:03.861244 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.861224    2559 scope.go:117] "RemoveContainer" containerID="da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"
Jul 28 17:13:03.861495 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:13:03.861475    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9\": container with ID starting with da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9 not found: ID does not exist" containerID="da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"
Jul 28 17:13:03.861536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.861505    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9"} err="failed to get container status \"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9\": rpc error: code = NotFound desc = could not find container \"da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9\": container with ID starting with da1c459298b9e0a29ff8cbf91e4012b4065b9a8717ced9544c9dc4bed99915c9 not found: ID does not exist"
Jul 28 17:13:03.877409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.877386    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:13:03.884809 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:03.884781    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f9b477fd4-l56m4"]
Jul 28 17:13:05.672659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:05.672623    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9621deaa-88dd-4cd9-a82b-701ac688602e" path="/var/lib/kubelet/pods/9621deaa-88dd-4cd9-a82b-701ac688602e/volumes"
Jul 28 17:13:47.011657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.011586    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"]
Jul 28 17:13:47.012123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.011909    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9621deaa-88dd-4cd9-a82b-701ac688602e" containerName="console"
Jul 28 17:13:47.012123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.011920    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9621deaa-88dd-4cd9-a82b-701ac688602e" containerName="console"
Jul 28 17:13:47.012123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.011992    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9621deaa-88dd-4cd9-a82b-701ac688602e" containerName="console"
Jul 28 17:13:47.013902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.013888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.016701 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.016670    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:13:47.016821 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.016700    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:13:47.016821 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.016750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6cmsw\""
Jul 28 17:13:47.027744 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.027723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"]
Jul 28 17:13:47.167332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.167300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.167502 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.167365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.167502 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.167388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5q8\" (UniqueName: \"kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.268197 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.268124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.268197 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.268186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.268361 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.268213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5q8\" (UniqueName: \"kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.268514 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.268494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.268588 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.268573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.277183 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.277154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5q8\" (UniqueName: \"kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.323158 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.323128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:13:47.449191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:47.449164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"]
Jul 28 17:13:47.451845 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:13:47.451815    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd87172_f761_49c7_a1ae_5803c4f9cbca.slice/crio-809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d WatchSource:0}: Error finding container 809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d: Status 404 returned error can't find the container with id 809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d
Jul 28 17:13:48.030640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:48.030594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx" event={"ID":"7dd87172-f761-49c7-a1ae-5803c4f9cbca","Type":"ContainerStarted","Data":"809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d"}
Jul 28 17:13:54.057235 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:54.057202    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerID="d9f5517645f7c32c1a9eef87c4c2c494c80857b2b6b7ab82de845cdf4f675b8b" exitCode=0
Jul 28 17:13:54.057640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:54.057286    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx" event={"ID":"7dd87172-f761-49c7-a1ae-5803c4f9cbca","Type":"ContainerDied","Data":"d9f5517645f7c32c1a9eef87c4c2c494c80857b2b6b7ab82de845cdf4f675b8b"}
Jul 28 17:13:57.072706 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:57.072670    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerID="95291c97e648dd15945dda97dec175f737831d70952b559b266ecacc7f0fcc52" exitCode=0
Jul 28 17:13:57.073097 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:13:57.072712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx" event={"ID":"7dd87172-f761-49c7-a1ae-5803c4f9cbca","Type":"ContainerDied","Data":"95291c97e648dd15945dda97dec175f737831d70952b559b266ecacc7f0fcc52"}
Jul 28 17:14:04.105087 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:04.105056    2559 generic.go:356] "Generic (PLEG): container finished" podID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerID="284f1356f4b6fb0c1f729d6ea234137dbb5724795e2924f6bc67fe858076e779" exitCode=0
Jul 28 17:14:04.105453 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:04.105103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx" event={"ID":"7dd87172-f761-49c7-a1ae-5803c4f9cbca","Type":"ContainerDied","Data":"284f1356f4b6fb0c1f729d6ea234137dbb5724795e2924f6bc67fe858076e779"}
Jul 28 17:14:05.248137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.248113    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:14:05.316362 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.316335    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp5q8\" (UniqueName: \"kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8\") pod \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") "
Jul 28 17:14:05.316495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.316407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util\") pod \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") "
Jul 28 17:14:05.316495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.316457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle\") pod \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\" (UID: \"7dd87172-f761-49c7-a1ae-5803c4f9cbca\") "
Jul 28 17:14:05.317066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.317042    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle" (OuterVolumeSpecName: "bundle") pod "7dd87172-f761-49c7-a1ae-5803c4f9cbca" (UID: "7dd87172-f761-49c7-a1ae-5803c4f9cbca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:14:05.318414 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.318391    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8" (OuterVolumeSpecName: "kube-api-access-zp5q8") pod "7dd87172-f761-49c7-a1ae-5803c4f9cbca" (UID: "7dd87172-f761-49c7-a1ae-5803c4f9cbca"). InnerVolumeSpecName "kube-api-access-zp5q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:14:05.320699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.320676    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util" (OuterVolumeSpecName: "util") pod "7dd87172-f761-49c7-a1ae-5803c4f9cbca" (UID: "7dd87172-f761-49c7-a1ae-5803c4f9cbca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:14:05.417386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.417315    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-util\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:14:05.417386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.417342    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dd87172-f761-49c7-a1ae-5803c4f9cbca-bundle\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:14:05.417386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:05.417355    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5q8\" (UniqueName: \"kubernetes.io/projected/7dd87172-f761-49c7-a1ae-5803c4f9cbca-kube-api-access-zp5q8\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:14:06.114482 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:06.114447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx" event={"ID":"7dd87172-f761-49c7-a1ae-5803c4f9cbca","Type":"ContainerDied","Data":"809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d"}
Jul 28 17:14:06.114655 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:06.114488    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5lprlx"
Jul 28 17:14:06.114655 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:06.114495    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809c6b84e9a0428a361d1d2339d2d8ea876121c2562d7e3ac23ba24f8ba5a83d"
Jul 28 17:14:09.394601 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394568    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"]
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394931    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="util"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394944    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="util"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394952    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="extract"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394957    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="extract"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394974    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="pull"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.394979    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="pull"
Jul 28 17:14:09.395073 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.395033    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd87172-f761-49c7-a1ae-5803c4f9cbca" containerName="extract"
Jul 28 17:14:09.398684 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.398668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.402314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.402292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-cc2x6\""
Jul 28 17:14:09.402426 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.402335    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:14:09.402426 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.402408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:14:09.402653 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.402637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:14:09.411069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.411048    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"]
Jul 28 17:14:09.449568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.449539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6d928048-fe65-4dc0-b14a-b17869ecb9e5-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.449568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.449566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gzs\" (UniqueName: \"kubernetes.io/projected/6d928048-fe65-4dc0-b14a-b17869ecb9e5-kube-api-access-w2gzs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.550518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.550491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6d928048-fe65-4dc0-b14a-b17869ecb9e5-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.550640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.550525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gzs\" (UniqueName: \"kubernetes.io/projected/6d928048-fe65-4dc0-b14a-b17869ecb9e5-kube-api-access-w2gzs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.552894 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.552853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6d928048-fe65-4dc0-b14a-b17869ecb9e5-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.561519 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.561498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gzs\" (UniqueName: \"kubernetes.io/projected/6d928048-fe65-4dc0-b14a-b17869ecb9e5-kube-api-access-w2gzs\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc\" (UID: \"6d928048-fe65-4dc0-b14a-b17869ecb9e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.708701 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.708624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:09.845606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:09.845581    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"]
Jul 28 17:14:09.847551 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:14:09.847521    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d928048_fe65_4dc0_b14a_b17869ecb9e5.slice/crio-cd6ed695b87331da4e8527cc694b442ffc0b7bef74925f70212e37063c8d6c3e WatchSource:0}: Error finding container cd6ed695b87331da4e8527cc694b442ffc0b7bef74925f70212e37063c8d6c3e: Status 404 returned error can't find the container with id cd6ed695b87331da4e8527cc694b442ffc0b7bef74925f70212e37063c8d6c3e
Jul 28 17:14:10.132534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:10.132503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc" event={"ID":"6d928048-fe65-4dc0-b14a-b17869ecb9e5","Type":"ContainerStarted","Data":"cd6ed695b87331da4e8527cc694b442ffc0b7bef74925f70212e37063c8d6c3e"}
Jul 28 17:14:14.152938 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:14.152905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc" event={"ID":"6d928048-fe65-4dc0-b14a-b17869ecb9e5","Type":"ContainerStarted","Data":"38286041f4043aee25dadc804106b23bbd03ce7c3207eb04d1837e9773f45992"}
Jul 28 17:14:14.153329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:14.152979    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:14:14.176657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:14.176609    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc" podStartSLOduration=1.80950034 podStartE2EDuration="5.176595706s" totalImagesPullingTime="3.367095366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:14.174651634 +0000 UTC m=+293.057058775" watchObservedRunningTime="2026-07-28 17:14:14.176595706 +0000 UTC m=+293.059002822"
Jul 28 17:14:21.488614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:21.488582    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:14:21.492151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:21.492130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:14:21.498203 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:21.498186    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:14:35.160652 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:14:35.160620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fx4zc"
Jul 28 17:15:32.573375 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.573330    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-7p44g"]
Jul 28 17:15:32.575446 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.575427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.577587 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.577558    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:15:32.577587 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.577571    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vd2g2\""
Jul 28 17:15:32.578401 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.578359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 17:15:32.578509 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.578429    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:15:32.587970 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.587948    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-7p44g"]
Jul 28 17:15:32.664191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.664153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7t5\" (UniqueName: \"kubernetes.io/projected/0cc56039-5d09-4043-a236-609346f1190c-kube-api-access-vn7t5\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.664356 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.664223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc56039-5d09-4043-a236-609346f1190c-cert\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.765572 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.765535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc56039-5d09-4043-a236-609346f1190c-cert\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.765741 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.765592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7t5\" (UniqueName: \"kubernetes.io/projected/0cc56039-5d09-4043-a236-609346f1190c-kube-api-access-vn7t5\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.767912 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.767887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cc56039-5d09-4043-a236-609346f1190c-cert\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.776676 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.776652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7t5\" (UniqueName: \"kubernetes.io/projected/0cc56039-5d09-4043-a236-609346f1190c-kube-api-access-vn7t5\") pod \"llmisvc-controller-manager-654c557856-7p44g\" (UID: \"0cc56039-5d09-4043-a236-609346f1190c\") " pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:32.887645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:32.887558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:33.039933 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:33.039900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-7p44g"]
Jul 28 17:15:33.043318 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:15:33.043285    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cc56039_5d09_4043_a236_609346f1190c.slice/crio-1a0516ac6eaf5ba631678f7238f71f14eea5359a35cfdb9d58164b9222a0a9f8 WatchSource:0}: Error finding container 1a0516ac6eaf5ba631678f7238f71f14eea5359a35cfdb9d58164b9222a0a9f8: Status 404 returned error can't find the container with id 1a0516ac6eaf5ba631678f7238f71f14eea5359a35cfdb9d58164b9222a0a9f8
Jul 28 17:15:33.044678 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:33.044660    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:15:33.503122 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:33.503090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-7p44g" event={"ID":"0cc56039-5d09-4043-a236-609346f1190c","Type":"ContainerStarted","Data":"1a0516ac6eaf5ba631678f7238f71f14eea5359a35cfdb9d58164b9222a0a9f8"}
Jul 28 17:15:35.513234 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:35.513199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-7p44g" event={"ID":"0cc56039-5d09-4043-a236-609346f1190c","Type":"ContainerStarted","Data":"facc6cea41950ac3b2826cc3ff2585c978c04aeb049cce6e82955aaa176cff7e"}
Jul 28 17:15:35.513646 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:35.513303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:15:35.542022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:15:35.541971    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-7p44g" podStartSLOduration=1.204538401 podStartE2EDuration="3.541956064s" totalImagesPullingTime="2.337417663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:35.539592204 +0000 UTC m=+374.421999361" watchObservedRunningTime="2026-07-28 17:15:35.541956064 +0000 UTC m=+374.424363178"
Jul 28 17:16:06.519419 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:06.519386    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-7p44g"
Jul 28 17:16:49.113430 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.113342    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dmz9n"]
Jul 28 17:16:49.115850 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.115829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.119598 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.119580    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:16:49.119598 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.119693    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ctsj9\""
Jul 28 17:16:49.127704 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.127681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dmz9n"]
Jul 28 17:16:49.132830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.132810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6c5l9"]
Jul 28 17:16:49.134815 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.134799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.141579 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.141557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6b5t6\""
Jul 28 17:16:49.141836 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.141820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:16:49.155109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.155086    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6c5l9"]
Jul 28 17:16:49.193359 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.193327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqh4\" (UniqueName: \"kubernetes.io/projected/bf4cf9fd-09a2-4c72-8463-4287f91edd29-kube-api-access-qrqh4\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.193518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.193365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4cf9fd-09a2-4c72-8463-4287f91edd29-cert\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.193518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.193420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhwl\" (UniqueName: \"kubernetes.io/projected/e5166ab4-8a8e-49d9-a712-030c67d2324f-kube-api-access-9lhwl\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.193518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.193488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5166ab4-8a8e-49d9-a712-030c67d2324f-tls-certs\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.294229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.294189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhwl\" (UniqueName: \"kubernetes.io/projected/e5166ab4-8a8e-49d9-a712-030c67d2324f-kube-api-access-9lhwl\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.294410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.294255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5166ab4-8a8e-49d9-a712-030c67d2324f-tls-certs\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.294410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.294295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqh4\" (UniqueName: \"kubernetes.io/projected/bf4cf9fd-09a2-4c72-8463-4287f91edd29-kube-api-access-qrqh4\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.294410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.294319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4cf9fd-09a2-4c72-8463-4287f91edd29-cert\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.296852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.296821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4cf9fd-09a2-4c72-8463-4287f91edd29-cert\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.296969 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.296821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5166ab4-8a8e-49d9-a712-030c67d2324f-tls-certs\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.306525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.306504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhwl\" (UniqueName: \"kubernetes.io/projected/e5166ab4-8a8e-49d9-a712-030c67d2324f-kube-api-access-9lhwl\") pod \"model-serving-api-7b954ffc95-dmz9n\" (UID: \"e5166ab4-8a8e-49d9-a712-030c67d2324f\") " pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.306638 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.306569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqh4\" (UniqueName: \"kubernetes.io/projected/bf4cf9fd-09a2-4c72-8463-4287f91edd29-kube-api-access-qrqh4\") pod \"odh-model-controller-75d9cdbddf-6c5l9\" (UID: \"bf4cf9fd-09a2-4c72-8463-4287f91edd29\") " pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.425758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.425664    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:49.446689 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.446655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:49.572077 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.572039    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dmz9n"]
Jul 28 17:16:49.574926 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:16:49.574890    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5166ab4_8a8e_49d9_a712_030c67d2324f.slice/crio-1a576fdeaab5848fb6617d1c630843805baecc5b3c4497763954fc69800cbaad WatchSource:0}: Error finding container 1a576fdeaab5848fb6617d1c630843805baecc5b3c4497763954fc69800cbaad: Status 404 returned error can't find the container with id 1a576fdeaab5848fb6617d1c630843805baecc5b3c4497763954fc69800cbaad
Jul 28 17:16:49.593923 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.593841    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6c5l9"]
Jul 28 17:16:49.596425 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:16:49.596400    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4cf9fd_09a2_4c72_8463_4287f91edd29.slice/crio-127a93708103dc6100d0073ebd5266a45647ecf538baf7712c0970f9e5048b25 WatchSource:0}: Error finding container 127a93708103dc6100d0073ebd5266a45647ecf538baf7712c0970f9e5048b25: Status 404 returned error can't find the container with id 127a93708103dc6100d0073ebd5266a45647ecf538baf7712c0970f9e5048b25
Jul 28 17:16:49.840725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.840693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9" event={"ID":"bf4cf9fd-09a2-4c72-8463-4287f91edd29","Type":"ContainerStarted","Data":"127a93708103dc6100d0073ebd5266a45647ecf538baf7712c0970f9e5048b25"}
Jul 28 17:16:49.841777 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:49.841757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dmz9n" event={"ID":"e5166ab4-8a8e-49d9-a712-030c67d2324f","Type":"ContainerStarted","Data":"1a576fdeaab5848fb6617d1c630843805baecc5b3c4497763954fc69800cbaad"}
Jul 28 17:16:53.863663 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.863627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dmz9n" event={"ID":"e5166ab4-8a8e-49d9-a712-030c67d2324f","Type":"ContainerStarted","Data":"d4b3829b766b4dd6d167460ee133496cefe679d6e335f1d4b1e267142e72fdae"}
Jul 28 17:16:53.864097 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.863745    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:16:53.865248 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.865227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9" event={"ID":"bf4cf9fd-09a2-4c72-8463-4287f91edd29","Type":"ContainerStarted","Data":"c9b4b94fc8f9fa620179cf28ca2a02059d04995ff04f8e6e6099e7bf88e809c1"}
Jul 28 17:16:53.865358 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.865344    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:16:53.882992 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.882952    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dmz9n" podStartSLOduration=1.1810943 podStartE2EDuration="4.882938883s" totalImagesPullingTime="3.701844583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:53.880789914 +0000 UTC m=+452.763197030" watchObservedRunningTime="2026-07-28 17:16:53.882938883 +0000 UTC m=+452.765346001"
Jul 28 17:16:53.898922 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:16:53.898862    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9" podStartSLOduration=1.226959826 podStartE2EDuration="4.898847559s" totalImagesPullingTime="3.671887733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:53.897136563 +0000 UTC m=+452.779543679" watchObservedRunningTime="2026-07-28 17:16:53.898847559 +0000 UTC m=+452.781254687"
Jul 28 17:17:04.871857 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:04.871824    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-6c5l9"
Jul 28 17:17:04.874074 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:04.874051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dmz9n"
Jul 28 17:17:30.345463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:30.345423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:17:55.371309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.371268    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84bd7f59f4-ngq8n" podUID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" containerName="console" containerID="cri-o://0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5" gracePeriod=15
Jul 28 17:17:55.616756 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.616735    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:17:55.645409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645348    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645514    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6v8q\" (UniqueName: \"kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645714 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645606    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645714 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.645714 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config\") pod \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\" (UID: \"eb2aab90-70f8-4956-8fe0-5c5f2d89937d\") "
Jul 28 17:17:55.646314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config" (OuterVolumeSpecName: "console-config") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:17:55.646314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.645967    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.646480 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.646439    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:17:55.646734 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.646612    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:17:55.646734 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.646708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:17:55.647968 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.647943    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:17:55.648067 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.647963    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:17:55.648067 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.647965    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q" (OuterVolumeSpecName: "kube-api-access-m6v8q") pod "eb2aab90-70f8-4956-8fe0-5c5f2d89937d" (UID: "eb2aab90-70f8-4956-8fe0-5c5f2d89937d"). InnerVolumeSpecName "kube-api-access-m6v8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:17:55.747165 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747137    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-oauth-serving-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.747165 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747160    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-trusted-ca-bundle\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.747165 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747171    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-service-ca\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.747355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747180    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-oauth-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.747355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747189    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-console-serving-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:55.747355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:55.747197    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6v8q\" (UniqueName: \"kubernetes.io/projected/eb2aab90-70f8-4956-8fe0-5c5f2d89937d-kube-api-access-m6v8q\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:17:56.146802 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.146772    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" containerID="0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5" exitCode=0
Jul 28 17:17:56.147022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.146832    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84bd7f59f4-ngq8n"
Jul 28 17:17:56.147022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.146841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd7f59f4-ngq8n" event={"ID":"eb2aab90-70f8-4956-8fe0-5c5f2d89937d","Type":"ContainerDied","Data":"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"}
Jul 28 17:17:56.147022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.146898    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84bd7f59f4-ngq8n" event={"ID":"eb2aab90-70f8-4956-8fe0-5c5f2d89937d","Type":"ContainerDied","Data":"d15707e0a5152d1b64a9986a991cdcefb0751cfaa6a5b832cd415a5b61ae5d38"}
Jul 28 17:17:56.147022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.146918    2559 scope.go:117] "RemoveContainer" containerID="0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"
Jul 28 17:17:56.155304 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.155288    2559 scope.go:117] "RemoveContainer" containerID="0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"
Jul 28 17:17:56.155552 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:17:56.155531    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5\": container with ID starting with 0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5 not found: ID does not exist" containerID="0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"
Jul 28 17:17:56.155637 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.155559    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5"} err="failed to get container status \"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5\": rpc error: code = NotFound desc = could not find container \"0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5\": container with ID starting with 0f7c28a7c61fe8086ee858f1b5a5310e6f591d994870794f61892f89c9ad5fb5 not found: ID does not exist"
Jul 28 17:17:56.168309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.168284    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:17:56.173830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:56.173811    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84bd7f59f4-ngq8n"]
Jul 28 17:17:57.673440 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:17:57.673411    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" path="/var/lib/kubelet/pods/eb2aab90-70f8-4956-8fe0-5c5f2d89937d/volumes"
Jul 28 17:18:39.067569 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.067530    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:18:39.068068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.067912    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" containerName="console"
Jul 28 17:18:39.068068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.067924    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" containerName="console"
Jul 28 17:18:39.068068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.067986    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2aab90-70f8-4956-8fe0-5c5f2d89937d" containerName="console"
Jul 28 17:18:39.072257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.072240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.074775 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.074753    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:18:39.074955 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.074894    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:18:39.075075 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.075035    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:18:39.075176 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.075103    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:18:39.075176 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.075126    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2ecea-kube-rbac-proxy-sar-config\""
Jul 28 17:18:39.075176 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.075156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:18:39.075751 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.075736    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-2ecea-predictor-serving-cert\""
Jul 28 17:18:39.082692 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.082668    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:18:39.198399 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.198355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvhr\" (UniqueName: \"kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.198563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.198442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.198563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.198480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.198563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.198516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.198725 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.198602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvhr\" (UniqueName: \"kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.299644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299623    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.300006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.299984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.300097 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.300023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.301614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.301588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.308307 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.308286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvhr\" (UniqueName: \"kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr\") pod \"isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.383081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.383020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:18:39.516489 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:39.516466    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:18:39.518382 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:18:39.518357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4daae4a_9173_48d3_aff8_78ad0256990d.slice/crio-b9f891e9044d8bd2944d92fc461fd26d982d7fa46a74c47d7415b93c5f6b9a44 WatchSource:0}: Error finding container b9f891e9044d8bd2944d92fc461fd26d982d7fa46a74c47d7415b93c5f6b9a44: Status 404 returned error can't find the container with id b9f891e9044d8bd2944d92fc461fd26d982d7fa46a74c47d7415b93c5f6b9a44
Jul 28 17:18:40.347945 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:40.347908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerStarted","Data":"b9f891e9044d8bd2944d92fc461fd26d982d7fa46a74c47d7415b93c5f6b9a44"}
Jul 28 17:18:43.367058 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:43.367022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerStarted","Data":"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"}
Jul 28 17:18:47.386415 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:47.386385    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerID="c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334" exitCode=0
Jul 28 17:18:47.386835 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:18:47.386427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerDied","Data":"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"}
Jul 28 17:19:01.472626 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:01.472548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerStarted","Data":"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"}
Jul 28 17:19:04.495710 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:04.495676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerStarted","Data":"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"}
Jul 28 17:19:06.508209 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:06.508175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerStarted","Data":"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"}
Jul 28 17:19:06.508573 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:06.508295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:19:06.540491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:06.540390    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podStartSLOduration=1.4610224170000001 podStartE2EDuration="27.540374621s" totalImagesPullingTime="22.116980193s" totalInitContainerRuntime="3.962372011s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:39 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:06.537745853 +0000 UTC m=+585.420152995" watchObservedRunningTime="2026-07-28 17:19:06.540374621 +0000 UTC m=+585.422781737"
Jul 28 17:19:07.512626 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:07.512590    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:19:07.513107 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:07.512665    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:19:07.513850 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:07.513814    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:07.514480 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:07.514458    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:08.516584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:08.516542    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:08.517085 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:08.517059    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:08.520440 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:08.520420    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:19:09.522246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:09.522209    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:09.522729 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:09.522548    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:19.523047 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:19.523006    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:19.523606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:19.523572    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:21.529343 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:21.529312    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:19:21.529795 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:21.529594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:19:29.522458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:29.522410    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:29.522929 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:29.522910    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:39.522838 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:39.522799    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:39.525460 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:39.523318    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:49.522418 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:49.522303    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:49.522850 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:49.522829    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:59.522801 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:59.522759    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:19:59.523358 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:19:59.523322    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:09.522618 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:09.522564    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:20:09.523151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:09.522993    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:16.667989 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:16.667960    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:16.668463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:16.668054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:24.017279 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.017243    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:20:24.017679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.017595    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" containerID="cri-o://74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266" gracePeriod=30
Jul 28 17:20:24.017748 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.017657    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" containerID="cri-o://d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4" gracePeriod=30
Jul 28 17:20:24.017800 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.017661    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" containerID="cri-o://a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26" gracePeriod=30
Jul 28 17:20:24.151421 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.151389    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:20:24.153901 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.153857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.156488 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.156468    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5fb11-predictor-serving-cert\""
Jul 28 17:20:24.156944 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.156924    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5fb11-kube-rbac-proxy-sar-config\""
Jul 28 17:20:24.166849 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.166778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:20:24.189141 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.189117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4nq\" (UniqueName: \"kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.189299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.189185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.189299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.189276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.189412 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.189307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.189412 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.189374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.247659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.247627    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:20:24.250292 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.250271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.252478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.252457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5fb11-predictor-serving-cert\""
Jul 28 17:20:24.252583 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.252566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-5fb11-kube-rbac-proxy-sar-config\""
Jul 28 17:20:24.261610 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.261588    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:20:24.290542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.290542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4nq\" (UniqueName: \"kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.290762 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.290762 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.290900 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.290900 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.291008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290900    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8865v\" (UniqueName: \"kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.291008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.291008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.291008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.290996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.291154 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:24.291103    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-serving-cert: secret "isvc-sklearn-graph-raw-5fb11-predictor-serving-cert" not found
Jul 28 17:20:24.291196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.291175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.291228 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:24.291191    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls podName:050d1d09-c39f-47ec-b83d-4c013ecfe8a4 nodeName:}" failed. No retries permitted until 2026-07-28 17:20:24.791170676 +0000 UTC m=+663.673577769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls") pod "isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4") : secret "isvc-sklearn-graph-raw-5fb11-predictor-serving-cert" not found
Jul 28 17:20:24.291267 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.291250    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.291304 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.291268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.300318 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.300296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4nq\" (UniqueName: \"kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.391956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.391927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392092 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.391968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8865v\" (UniqueName: \"kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392261 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392555 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392662 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.392779 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.392692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.394518 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.394495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.400945 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.400892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8865v\" (UniqueName: \"kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v\") pod \"isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.560547 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.560517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:24.693221 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.693191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:20:24.695530 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:20:24.695506    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679b1732_db62_4380_b274_b79ad4de310c.slice/crio-0b452582cee3c460f81c0aea1c572ba81146038778ff81da36e7adb3978064d8 WatchSource:0}: Error finding container 0b452582cee3c460f81c0aea1c572ba81146038778ff81da36e7adb3978064d8: Status 404 returned error can't find the container with id 0b452582cee3c460f81c0aea1c572ba81146038778ff81da36e7adb3978064d8
Jul 28 17:20:24.794928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.794895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.797034 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.797017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") pod \"isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:24.862960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.862848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerStarted","Data":"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"}
Jul 28 17:20:24.862960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.862947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerStarted","Data":"0b452582cee3c460f81c0aea1c572ba81146038778ff81da36e7adb3978064d8"}
Jul 28 17:20:24.865383 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.865349    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerID="a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26" exitCode=2
Jul 28 17:20:24.865494 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:24.865397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerDied","Data":"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"}
Jul 28 17:20:25.065545 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:25.065513    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:25.204850 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:25.204830    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:20:25.207775 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:20:25.207749    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050d1d09_c39f_47ec_b83d_4c013ecfe8a4.slice/crio-ca3be6e76cb0e6ba0fafe8d503b495d916f43e9fe95e2055210f96169863d232 WatchSource:0}: Error finding container ca3be6e76cb0e6ba0fafe8d503b495d916f43e9fe95e2055210f96169863d232: Status 404 returned error can't find the container with id ca3be6e76cb0e6ba0fafe8d503b495d916f43e9fe95e2055210f96169863d232
Jul 28 17:20:25.871255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:25.871220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerStarted","Data":"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"}
Jul 28 17:20:25.871432 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:25.871281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerStarted","Data":"ca3be6e76cb0e6ba0fafe8d503b495d916f43e9fe95e2055210f96169863d232"}
Jul 28 17:20:26.668150 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:26.668111    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:20:26.668587 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:26.668425    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:28.517548 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:28.517500    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:28.886564 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:28.886531    2559 generic.go:356] "Generic (PLEG): container finished" podID="679b1732-db62-4380-b274-b79ad4de310c" containerID="68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc" exitCode=0
Jul 28 17:20:28.886740 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:28.886603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerDied","Data":"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"}
Jul 28 17:20:28.889469 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:28.889450    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerID="74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266" exitCode=0
Jul 28 17:20:28.889561 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:28.889498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerDied","Data":"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"}
Jul 28 17:20:29.895575 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:29.895544    2559 generic.go:356] "Generic (PLEG): container finished" podID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerID="92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f" exitCode=0
Jul 28 17:20:29.896049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:29.895616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerDied","Data":"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"}
Jul 28 17:20:31.911473 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:31.911431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerStarted","Data":"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"}
Jul 28 17:20:31.911935 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:31.911520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerStarted","Data":"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"}
Jul 28 17:20:31.911935 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:31.911760    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:31.932668 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:31.932610    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podStartSLOduration=4.241934657 podStartE2EDuration="7.932588992s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690654335s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:31.931530851 +0000 UTC m=+670.813937967" watchObservedRunningTime="2026-07-28 17:20:31.932588992 +0000 UTC m=+670.814996109"
Jul 28 17:20:32.916345 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:32.916310    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:32.917610 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:32.917581    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:20:33.517223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:33.517184    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:33.922260 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:33.922218    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:20:36.668305 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:36.668257    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:20:36.668747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:36.668666    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:38.516943 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:38.516863    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:38.517405 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:38.517038    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:38.928177 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:38.928149    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:20:38.928681 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:38.928647    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:20:43.517488 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:43.517448    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:46.667584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:46.667530    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:20:46.668076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:46.667675    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:46.668076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:46.668033    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:46.668204 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:46.668152    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:47.999898 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:47.999842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerStarted","Data":"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"}
Jul 28 17:20:48.000370 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:47.999932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerStarted","Data":"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"}
Jul 28 17:20:48.000370 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:48.000159    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:48.020692 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:48.020646    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podStartSLOduration=1.741882269 podStartE2EDuration="24.020632127s" totalImagesPullingTime="18.450275375s" totalInitContainerRuntime="3.828474483s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:48.019128363 +0000 UTC m=+686.901535480" watchObservedRunningTime="2026-07-28 17:20:48.020632127 +0000 UTC m=+686.903039242"
Jul 28 17:20:48.517768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:48.517725    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:48.928942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:48.928903    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:20:49.005241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:49.005212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:49.006675 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:49.006649    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:20:50.009925 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:50.009890    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:20:53.516930 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:53.516892    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:20:54.732744 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.732717    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:54.868161 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868074    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert\") pod \"d4daae4a-9173-48d3-aff8-78ad0256990d\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") "
Jul 28 17:20:54.868161 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868131    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xvhr\" (UniqueName: \"kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr\") pod \"d4daae4a-9173-48d3-aff8-78ad0256990d\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") "
Jul 28 17:20:54.868161 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868160    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config\") pod \"d4daae4a-9173-48d3-aff8-78ad0256990d\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") "
Jul 28 17:20:54.868430 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868227    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls\") pod \"d4daae4a-9173-48d3-aff8-78ad0256990d\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") "
Jul 28 17:20:54.868430 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868346    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location\") pod \"d4daae4a-9173-48d3-aff8-78ad0256990d\" (UID: \"d4daae4a-9173-48d3-aff8-78ad0256990d\") "
Jul 28 17:20:54.868546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4daae4a-9173-48d3-aff8-78ad0256990d" (UID: "d4daae4a-9173-48d3-aff8-78ad0256990d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:54.868633 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868597    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4daae4a-9173-48d3-aff8-78ad0256990d" (UID: "d4daae4a-9173-48d3-aff8-78ad0256990d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:54.868688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868660    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4daae4a-9173-48d3-aff8-78ad0256990d" (UID: "d4daae4a-9173-48d3-aff8-78ad0256990d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:20:54.868790 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868774    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:20:54.868834 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868795    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:20:54.868834 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.868805    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4daae4a-9173-48d3-aff8-78ad0256990d-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:20:54.870462 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.870439    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4daae4a-9173-48d3-aff8-78ad0256990d" (UID: "d4daae4a-9173-48d3-aff8-78ad0256990d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:54.870534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.870500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr" (OuterVolumeSpecName: "kube-api-access-8xvhr") pod "d4daae4a-9173-48d3-aff8-78ad0256990d" (UID: "d4daae4a-9173-48d3-aff8-78ad0256990d"). InnerVolumeSpecName "kube-api-access-8xvhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:54.969625 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.969597    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4daae4a-9173-48d3-aff8-78ad0256990d-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:20:54.969625 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:54.969623    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvhr\" (UniqueName: \"kubernetes.io/projected/d4daae4a-9173-48d3-aff8-78ad0256990d-kube-api-access-8xvhr\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:20:55.014559 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.014533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:20:55.015035 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.015004    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:20:55.037069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.037035    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerID="d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4" exitCode=0
Jul 28 17:20:55.037206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.037105    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"
Jul 28 17:20:55.037206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.037113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerDied","Data":"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"}
Jul 28 17:20:55.037206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.037183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s" event={"ID":"d4daae4a-9173-48d3-aff8-78ad0256990d","Type":"ContainerDied","Data":"b9f891e9044d8bd2944d92fc461fd26d982d7fa46a74c47d7415b93c5f6b9a44"}
Jul 28 17:20:55.037308 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.037213    2559 scope.go:117] "RemoveContainer" containerID="d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"
Jul 28 17:20:55.046661 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.046644    2559 scope.go:117] "RemoveContainer" containerID="a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"
Jul 28 17:20:55.054213 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.054195    2559 scope.go:117] "RemoveContainer" containerID="74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"
Jul 28 17:20:55.061601 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.061569    2559 scope.go:117] "RemoveContainer" containerID="c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"
Jul 28 17:20:55.063454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.063436    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:20:55.069963 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.069945    2559 scope.go:117] "RemoveContainer" containerID="d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"
Jul 28 17:20:55.070245 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:55.070221    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4\": container with ID starting with d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4 not found: ID does not exist" containerID="d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"
Jul 28 17:20:55.070338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070255    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4"} err="failed to get container status \"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4\": rpc error: code = NotFound desc = could not find container \"d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4\": container with ID starting with d9e2cbef60069c855287f99bbf7932954cdef2a772941fb32928b661a768e2c4 not found: ID does not exist"
Jul 28 17:20:55.070338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070278    2559 scope.go:117] "RemoveContainer" containerID="a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"
Jul 28 17:20:55.070520 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:55.070505    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26\": container with ID starting with a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26 not found: ID does not exist" containerID="a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"
Jul 28 17:20:55.070558 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070526    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26"} err="failed to get container status \"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26\": rpc error: code = NotFound desc = could not find container \"a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26\": container with ID starting with a2cc78422a30ead1a265aafc50a0f751c6cd4a893c9b0b57403b25f2a1b17b26 not found: ID does not exist"
Jul 28 17:20:55.070558 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070539    2559 scope.go:117] "RemoveContainer" containerID="74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"
Jul 28 17:20:55.070640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070577    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s"]
Jul 28 17:20:55.070788 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:55.070772    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266\": container with ID starting with 74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266 not found: ID does not exist" containerID="74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"
Jul 28 17:20:55.070832 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070793    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266"} err="failed to get container status \"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266\": rpc error: code = NotFound desc = could not find container \"74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266\": container with ID starting with 74a96078d14a5a81dd5723c633335544938031e257c2317ab72be98fe917d266 not found: ID does not exist"
Jul 28 17:20:55.070832 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.070809    2559 scope.go:117] "RemoveContainer" containerID="c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"
Jul 28 17:20:55.071064 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:20:55.071048    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334\": container with ID starting with c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334 not found: ID does not exist" containerID="c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"
Jul 28 17:20:55.071116 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.071068    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334"} err="failed to get container status \"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334\": rpc error: code = NotFound desc = could not find container \"c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334\": container with ID starting with c014777d62ab8fc7c58d29717164b7ae78751ae2e4adaeef2dd8f07d44120334 not found: ID does not exist"
Jul 28 17:20:55.672381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:55.672347    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" path="/var/lib/kubelet/pods/d4daae4a-9173-48d3-aff8-78ad0256990d/volumes"
Jul 28 17:20:58.928719 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:20:58.928679    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:21:05.015642 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:05.015602    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:21:08.928892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:08.928834    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:21:15.015368 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:15.015278    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:21:18.930700 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:18.930655    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:21:25.015084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:25.015043    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:21:28.928965 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:28.928922    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:21:35.015721 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:35.015677    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:21:38.929240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:38.929201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:21:45.015786 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:45.015746    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:21:48.929718 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:48.929686    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:21:55.015773 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:21:55.015743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:22:14.409679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.409642    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:22:14.410175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.409956    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" containerID="cri-o://f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674" gracePeriod=30
Jul 28 17:22:14.410175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.410047    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kube-rbac-proxy" containerID="cri-o://77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c" gracePeriod=30
Jul 28 17:22:14.465445 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465418    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:22:14.465785 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465774    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container"
Jul 28 17:22:14.465829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465788    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container"
Jul 28 17:22:14.465829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465803    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="storage-initializer"
Jul 28 17:22:14.465829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465809    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="storage-initializer"
Jul 28 17:22:14.465829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465815    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy"
Jul 28 17:22:14.465829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465822    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy"
Jul 28 17:22:14.466024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465831    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent"
Jul 28 17:22:14.466024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465837    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent"
Jul 28 17:22:14.466024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465930    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kserve-container"
Jul 28 17:22:14.466024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465938    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="agent"
Jul 28 17:22:14.466024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.465946    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4daae4a-9173-48d3-aff8-78ad0256990d" containerName="kube-rbac-proxy"
Jul 28 17:22:14.468072 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.468057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.470326 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.470306    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c3acd-predictor-serving-cert\""
Jul 28 17:22:14.470457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.470420    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c3acd-kube-rbac-proxy-sar-config\""
Jul 28 17:22:14.480261 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.480238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:22:14.548278 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.548249    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:22:14.550680 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.550665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.551928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.551900    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.552062 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.551975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.552062 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.552021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.552062 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.552056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdmj\" (UniqueName: \"kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.552219 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.552090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.552942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.552917    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c3acd-kube-rbac-proxy-sar-config\""
Jul 28 17:22:14.553055 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.552953    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c3acd-predictor-serving-cert\""
Jul 28 17:22:14.562374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.562353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:22:14.653338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.653536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.653536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk29q\" (UniqueName: \"kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.653536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.653536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.653536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.653809 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.653809 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdmj\" (UniqueName: \"kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.653809 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.653809 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.653668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.654186 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.654163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.654465 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.654392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.654685 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.654668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.656574 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.656547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.663532 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.663471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdmj\" (UniqueName: \"kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj\") pod \"isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.678163 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.678137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:22:14.678371 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.678340    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" containerID="cri-o://e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea" gracePeriod=30
Jul 28 17:22:14.678484 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.678368    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy" containerID="cri-o://68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c" gracePeriod=30
Jul 28 17:22:14.754246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.754389 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk29q\" (UniqueName: \"kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.754389 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.754389 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.754389 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.754587 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:14.754417    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-c3acd-predictor-serving-cert" not found
Jul 28 17:22:14.754587 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:14.754481    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls podName:4009dc87-5434-45df-9454-e6bc4d10b70b nodeName:}" failed. No retries permitted until 2026-07-28 17:22:15.25446129 +0000 UTC m=+774.136868410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b") : secret "isvc-xgboost-graph-raw-hpa-c3acd-predictor-serving-cert" not found
Jul 28 17:22:14.754812 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.755005 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.754985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.755104 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.755081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.765307 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.765287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk29q\" (UniqueName: \"kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:14.781194 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.781167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:14.916454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:14.916419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:22:14.918605 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:22:14.918575    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e18970d_c15c_4ec7_9807_e23fc4a172f0.slice/crio-1e76d710b73f45e2f1a0debe3b6a1047e5e5ab00844f4b3172b75a724be464f2 WatchSource:0}: Error finding container 1e76d710b73f45e2f1a0debe3b6a1047e5e5ab00844f4b3172b75a724be464f2: Status 404 returned error can't find the container with id 1e76d710b73f45e2f1a0debe3b6a1047e5e5ab00844f4b3172b75a724be464f2
Jul 28 17:22:15.010769 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.010733    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 28 17:22:15.015104 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.015076    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:22:15.258831 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.258744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:15.261080 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.261051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:15.420835 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.420799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerStarted","Data":"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"}
Jul 28 17:22:15.421322 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.420917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerStarted","Data":"1e76d710b73f45e2f1a0debe3b6a1047e5e5ab00844f4b3172b75a724be464f2"}
Jul 28 17:22:15.423408 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.423379    2559 generic.go:356] "Generic (PLEG): container finished" podID="679b1732-db62-4380-b274-b79ad4de310c" containerID="68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c" exitCode=2
Jul 28 17:22:15.423555 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.423443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerDied","Data":"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"}
Jul 28 17:22:15.425910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.425890    2559 generic.go:356] "Generic (PLEG): container finished" podID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerID="77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c" exitCode=2
Jul 28 17:22:15.426026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.425921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerDied","Data":"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"}
Jul 28 17:22:15.460858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.460823    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:15.598567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:15.598536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:22:15.601002 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:22:15.600975    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4009dc87_5434_45df_9454_e6bc4d10b70b.slice/crio-21de5fd9a24b580d620ec9ff1e2556668857f38ec91f519a48a216ff45f799a5 WatchSource:0}: Error finding container 21de5fd9a24b580d620ec9ff1e2556668857f38ec91f519a48a216ff45f799a5: Status 404 returned error can't find the container with id 21de5fd9a24b580d620ec9ff1e2556668857f38ec91f519a48a216ff45f799a5
Jul 28 17:22:16.432361 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:16.432325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerStarted","Data":"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"}
Jul 28 17:22:16.432848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:16.432383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerStarted","Data":"21de5fd9a24b580d620ec9ff1e2556668857f38ec91f519a48a216ff45f799a5"}
Jul 28 17:22:18.922584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:18.922549    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 17:22:18.929359 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:18.929322    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:22:19.060109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.060087    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:22:19.088000 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.087938    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert\") pod \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") "
Jul 28 17:22:19.088127 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088045    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config\") pod \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") "
Jul 28 17:22:19.088127 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088081    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location\") pod \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") "
Jul 28 17:22:19.088233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") pod \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") "
Jul 28 17:22:19.088281 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088250    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm4nq\" (UniqueName: \"kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq\") pod \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\" (UID: \"050d1d09-c39f-47ec-b83d-4c013ecfe8a4\") "
Jul 28 17:22:19.088339 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088313    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "050d1d09-c39f-47ec-b83d-4c013ecfe8a4" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:19.088561 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.088525    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "050d1d09-c39f-47ec-b83d-4c013ecfe8a4" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:22:19.089044 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.089015    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "050d1d09-c39f-47ec-b83d-4c013ecfe8a4" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:19.089228 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.089212    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:19.089292 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.089232    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:19.089292 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.089249    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:19.090537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.090514    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "050d1d09-c39f-47ec-b83d-4c013ecfe8a4" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:22:19.090537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.090528    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq" (OuterVolumeSpecName: "kube-api-access-lm4nq") pod "050d1d09-c39f-47ec-b83d-4c013ecfe8a4" (UID: "050d1d09-c39f-47ec-b83d-4c013ecfe8a4"). InnerVolumeSpecName "kube-api-access-lm4nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:22:19.190191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.190162    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:19.190191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.190188    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4nq\" (UniqueName: \"kubernetes.io/projected/050d1d09-c39f-47ec-b83d-4c013ecfe8a4-kube-api-access-lm4nq\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:19.447175 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.447143    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerID="6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d" exitCode=0
Jul 28 17:22:19.447337 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.447215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerDied","Data":"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"}
Jul 28 17:22:19.448884 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.448849    2559 generic.go:356] "Generic (PLEG): container finished" podID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerID="036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42" exitCode=0
Jul 28 17:22:19.448985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.448927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerDied","Data":"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"}
Jul 28 17:22:19.450953 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.450932    2559 generic.go:356] "Generic (PLEG): container finished" podID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerID="f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674" exitCode=0
Jul 28 17:22:19.451050 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.450962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerDied","Data":"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"}
Jul 28 17:22:19.451050 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.450994    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"
Jul 28 17:22:19.451050 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.451012    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r" event={"ID":"050d1d09-c39f-47ec-b83d-4c013ecfe8a4","Type":"ContainerDied","Data":"ca3be6e76cb0e6ba0fafe8d503b495d916f43e9fe95e2055210f96169863d232"}
Jul 28 17:22:19.451050 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.451038    2559 scope.go:117] "RemoveContainer" containerID="77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"
Jul 28 17:22:19.462082 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.462059    2559 scope.go:117] "RemoveContainer" containerID="f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"
Jul 28 17:22:19.472117 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.472100    2559 scope.go:117] "RemoveContainer" containerID="92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"
Jul 28 17:22:19.483324 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.483303    2559 scope.go:117] "RemoveContainer" containerID="77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"
Jul 28 17:22:19.483621 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:19.483603    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c\": container with ID starting with 77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c not found: ID does not exist" containerID="77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"
Jul 28 17:22:19.483682 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.483628    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c"} err="failed to get container status \"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c\": rpc error: code = NotFound desc = could not find container \"77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c\": container with ID starting with 77ec477012459623856a4c63d23ceedefddb1afd949ae991b118cf4224e2fe5c not found: ID does not exist"
Jul 28 17:22:19.483682 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.483644    2559 scope.go:117] "RemoveContainer" containerID="f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"
Jul 28 17:22:19.483941 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:19.483864    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674\": container with ID starting with f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674 not found: ID does not exist" containerID="f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"
Jul 28 17:22:19.483983 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.483955    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674"} err="failed to get container status \"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674\": rpc error: code = NotFound desc = could not find container \"f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674\": container with ID starting with f1507336dc57cb12b9af0248e660a65cef4a96fab2efeea2c6b88208c9b1f674 not found: ID does not exist"
Jul 28 17:22:19.484027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.483980    2559 scope.go:117] "RemoveContainer" containerID="92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"
Jul 28 17:22:19.484239 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:19.484216    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f\": container with ID starting with 92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f not found: ID does not exist" containerID="92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"
Jul 28 17:22:19.484282 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.484248    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f"} err="failed to get container status \"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f\": rpc error: code = NotFound desc = could not find container \"92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f\": container with ID starting with 92343fdb05210f9a4ea86f9bb7af7d5ca9ac68abf02f8c33a4594b1db4ec713f not found: ID does not exist"
Jul 28 17:22:19.504569 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.504549    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:22:19.508444 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.508422    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r"]
Jul 28 17:22:19.673484 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:19.673407    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" path="/var/lib/kubelet/pods/050d1d09-c39f-47ec-b83d-4c013ecfe8a4/volumes"
Jul 28 17:22:20.010667 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.010575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 28 17:22:20.460602 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.460565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerStarted","Data":"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"}
Jul 28 17:22:20.460812 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.460630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerStarted","Data":"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"}
Jul 28 17:22:20.460909 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.460827    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:20.462717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.462690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerStarted","Data":"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"}
Jul 28 17:22:20.462811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.462745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerStarted","Data":"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"}
Jul 28 17:22:20.462987 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.462973    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:20.481196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.481145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podStartSLOduration=2.955034312 podStartE2EDuration="6.481132402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.52609809s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:20.478356659 +0000 UTC m=+779.360763775" watchObservedRunningTime="2026-07-28 17:22:20.481132402 +0000 UTC m=+779.363539518"
Jul 28 17:22:20.499240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:20.499204    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podStartSLOduration=2.785351604 podStartE2EDuration="6.499192521s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713840917s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:20.497747194 +0000 UTC m=+779.380154310" watchObservedRunningTime="2026-07-28 17:22:20.499192521 +0000 UTC m=+779.381599637"
Jul 28 17:22:21.468913 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.468863    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:21.469339 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.468946    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:21.469985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.469958    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:21.469985 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.469975    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:22:21.846598 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.846568    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:22:21.912791 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.912761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location\") pod \"679b1732-db62-4380-b274-b79ad4de310c\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") "
Jul 28 17:22:21.912791 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.912792    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config\") pod \"679b1732-db62-4380-b274-b79ad4de310c\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") "
Jul 28 17:22:21.912984 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.912831    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert\") pod \"679b1732-db62-4380-b274-b79ad4de310c\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") "
Jul 28 17:22:21.912984 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.912973    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls\") pod \"679b1732-db62-4380-b274-b79ad4de310c\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") "
Jul 28 17:22:21.913061 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8865v\" (UniqueName: \"kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v\") pod \"679b1732-db62-4380-b274-b79ad4de310c\" (UID: \"679b1732-db62-4380-b274-b79ad4de310c\") "
Jul 28 17:22:21.913151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "679b1732-db62-4380-b274-b79ad4de310c" (UID: "679b1732-db62-4380-b274-b79ad4de310c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:22:21.913230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "679b1732-db62-4380-b274-b79ad4de310c" (UID: "679b1732-db62-4380-b274-b79ad4de310c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:21.913230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913172    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "679b1732-db62-4380-b274-b79ad4de310c" (UID: "679b1732-db62-4380-b274-b79ad4de310c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:21.913342 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913272    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/679b1732-db62-4380-b274-b79ad4de310c-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:21.913342 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913284    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:21.913342 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.913292    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/679b1732-db62-4380-b274-b79ad4de310c-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:21.914805 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.914786    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "679b1732-db62-4380-b274-b79ad4de310c" (UID: "679b1732-db62-4380-b274-b79ad4de310c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:22:21.915007 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:21.914984    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v" (OuterVolumeSpecName: "kube-api-access-8865v") pod "679b1732-db62-4380-b274-b79ad4de310c" (UID: "679b1732-db62-4380-b274-b79ad4de310c"). InnerVolumeSpecName "kube-api-access-8865v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:22:22.013794 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.013711    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8865v\" (UniqueName: \"kubernetes.io/projected/679b1732-db62-4380-b274-b79ad4de310c-kube-api-access-8865v\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:22.013794 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.013744    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/679b1732-db62-4380-b274-b79ad4de310c-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:22:22.474947 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.474916    2559 generic.go:356] "Generic (PLEG): container finished" podID="679b1732-db62-4380-b274-b79ad4de310c" containerID="e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea" exitCode=0
Jul 28 17:22:22.475355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.474985    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"
Jul 28 17:22:22.475355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.475002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerDied","Data":"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"}
Jul 28 17:22:22.475355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.475065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt" event={"ID":"679b1732-db62-4380-b274-b79ad4de310c","Type":"ContainerDied","Data":"0b452582cee3c460f81c0aea1c572ba81146038778ff81da36e7adb3978064d8"}
Jul 28 17:22:22.475355 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.475095    2559 scope.go:117] "RemoveContainer" containerID="68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"
Jul 28 17:22:22.475688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.475665    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:22.475777 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.475665    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:22:22.483790 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.483770    2559 scope.go:117] "RemoveContainer" containerID="e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"
Jul 28 17:22:22.491352 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.491333    2559 scope.go:117] "RemoveContainer" containerID="68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"
Jul 28 17:22:22.497154 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.497131    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:22:22.499081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.499036    2559 scope.go:117] "RemoveContainer" containerID="68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:22.499372    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c\": container with ID starting with 68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c not found: ID does not exist" containerID="68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.499409    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c"} err="failed to get container status \"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c\": rpc error: code = NotFound desc = could not find container \"68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c\": container with ID starting with 68f3e156607b1c5e1d3ca29137b6909abe30f0aa3dad18c6c8a9e248140e507c not found: ID does not exist"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.499430    2559 scope.go:117] "RemoveContainer" containerID="e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:22.499744    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea\": container with ID starting with e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea not found: ID does not exist" containerID="e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.499772    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea"} err="failed to get container status \"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea\": rpc error: code = NotFound desc = could not find container \"e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea\": container with ID starting with e2e918a40d76fdc3d52b02d6be4c128264ab1f6fab3a9f1864a9204576b7b0ea not found: ID does not exist"
Jul 28 17:22:22.499810 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.499809    2559 scope.go:117] "RemoveContainer" containerID="68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"
Jul 28 17:22:22.500271 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:22:22.500074    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc\": container with ID starting with 68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc not found: ID does not exist" containerID="68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"
Jul 28 17:22:22.500271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.500101    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc"} err="failed to get container status \"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc\": rpc error: code = NotFound desc = could not find container \"68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc\": container with ID starting with 68d82a5ad627b7d604dc077ef0651949156ef9e3347847fd32b21a54151fbbcc not found: ID does not exist"
Jul 28 17:22:22.501672 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:22.501655    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt"]
Jul 28 17:22:23.673007 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:23.672969    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="679b1732-db62-4380-b274-b79ad4de310c" path="/var/lib/kubelet/pods/679b1732-db62-4380-b274-b79ad4de310c/volumes"
Jul 28 17:22:27.480232 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:27.480202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:22:27.480652 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:27.480287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:22:27.480706 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:27.480680    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:22:27.480771 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:27.480750    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:37.480639 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:37.480602    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:22:37.480639 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:37.480620    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:47.480741 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:47.480653    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:22:47.480741 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:47.480653    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:57.480825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:57.480789    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:22:57.481249 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:22:57.480795    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:23:07.481495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:07.481448    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:23:07.481983 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:07.481448    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:23:17.480801 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:17.480748    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:23:17.481249 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:17.480848    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:23:27.480713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:27.480666    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:23:27.481177 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:27.481030    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:23:37.481623 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:23:37.481594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:24:04.851654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.851619    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852037    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852049    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852059    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852064    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852079    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kube-rbac-proxy"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852084    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kube-rbac-proxy"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852091    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852096    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852125    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="storage-initializer"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="storage-initializer"
Jul 28 17:24:04.852136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852140    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="storage-initializer"
Jul 28 17:24:04.852457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852145    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="storage-initializer"
Jul 28 17:24:04.852457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852208    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kserve-container"
Jul 28 17:24:04.852457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852216    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kserve-container"
Jul 28 17:24:04.852457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852222    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="050d1d09-c39f-47ec-b83d-4c013ecfe8a4" containerName="kube-rbac-proxy"
Jul 28 17:24:04.852457 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.852229    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="679b1732-db62-4380-b274-b79ad4de310c" containerName="kube-rbac-proxy"
Jul 28 17:24:04.855436 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.855418    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.859692 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.859678    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-b8148-predictor-serving-cert\""
Jul 28 17:24:04.859748 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.859689    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-b8148-kube-rbac-proxy-sar-config\""
Jul 28 17:24:04.865066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.865040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.865193 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.865177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.865256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.865208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnq9\" (UniqueName: \"kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.866598 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.866577    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:24:04.915250 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.915224    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:24:04.915548 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.915512    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kube-rbac-proxy" containerID="cri-o://f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb" gracePeriod=30
Jul 28 17:24:04.915679 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.915507    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" containerID="cri-o://bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498" gracePeriod=30
Jul 28 17:24:04.964974 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.964944    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:24:04.965205 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.965169    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" containerID="cri-o://61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed" gracePeriod=30
Jul 28 17:24:04.965295 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.965251    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kube-rbac-proxy" containerID="cri-o://86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49" gracePeriod=30
Jul 28 17:24:04.965642 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.965593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.965708 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.965686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.965765 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.965716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnq9\" (UniqueName: \"kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.965932 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:04.965907    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-serving-cert: secret "message-dumper-raw-b8148-predictor-serving-cert" not found
Jul 28 17:24:04.966039 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:04.966008    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls podName:6f282bca-8e26-4e97-b1a4-0f193ac3ec56 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:05.465965789 +0000 UTC m=+884.348372889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls") pod "message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" (UID: "6f282bca-8e26-4e97-b1a4-0f193ac3ec56") : secret "message-dumper-raw-b8148-predictor-serving-cert" not found
Jul 28 17:24:04.966659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.966586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:04.977556 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:04.977533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnq9\" (UniqueName: \"kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:05.470906 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.470862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:05.473276 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.473254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") pod \"message-dumper-raw-b8148-predictor-86f78f7649-2xqrx\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") " pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:05.766193 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.766098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:05.904709 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.904686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:24:05.907144 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:24:05.907119    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f282bca_8e26_4e97_b1a4_0f193ac3ec56.slice/crio-8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd WatchSource:0}: Error finding container 8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd: Status 404 returned error can't find the container with id 8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd
Jul 28 17:24:05.908969 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.908950    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:24:05.971651 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.971622    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerID="f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb" exitCode=2
Jul 28 17:24:05.971786 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.971701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerDied","Data":"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"}
Jul 28 17:24:05.973537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.973518    2559 generic.go:356] "Generic (PLEG): container finished" podID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerID="86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49" exitCode=2
Jul 28 17:24:05.973634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.973590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerDied","Data":"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"}
Jul 28 17:24:05.974619 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:05.974597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerStarted","Data":"8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd"}
Jul 28 17:24:07.476521 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.476482    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:24:07.476938 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.476482    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 28 17:24:07.480772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.480749    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:24:07.480907 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.480784    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:24:07.988335 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.988301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerStarted","Data":"766b9fc522278c01d5dd1fcbe5ea3f05a4bd8e06745b2b65bcb6f43a966d2d64"}
Jul 28 17:24:07.988515 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.988364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerStarted","Data":"2ba26293fe79bc5817cc54f0eb6d5e05e118ba3d788895b1e0b25105b70fbccc"}
Jul 28 17:24:07.988515 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:07.988450    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:08.013049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.012998    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" podStartSLOduration=2.877668258 podStartE2EDuration="4.012983098s" totalImagesPullingTime="1.13531484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:08.011098501 +0000 UTC m=+886.893505629" watchObservedRunningTime="2026-07-28 17:24:08.012983098 +0000 UTC m=+886.895390213"
Jul 28 17:24:08.609486 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.609461    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:24:08.697053 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.696988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location\") pod \"4009dc87-5434-45df-9454-e6bc4d10b70b\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") "
Jul 28 17:24:08.697053 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") pod \"4009dc87-5434-45df-9454-e6bc4d10b70b\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") "
Jul 28 17:24:08.697053 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config\") pod \"4009dc87-5434-45df-9454-e6bc4d10b70b\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") "
Jul 28 17:24:08.697323 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697071    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk29q\" (UniqueName: \"kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q\") pod \"4009dc87-5434-45df-9454-e6bc4d10b70b\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") "
Jul 28 17:24:08.697323 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697120    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert\") pod \"4009dc87-5434-45df-9454-e6bc4d10b70b\" (UID: \"4009dc87-5434-45df-9454-e6bc4d10b70b\") "
Jul 28 17:24:08.697433 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4009dc87-5434-45df-9454-e6bc4d10b70b" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:08.697511 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4009dc87-5434-45df-9454-e6bc4d10b70b" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:08.697590 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.697555    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4009dc87-5434-45df-9454-e6bc4d10b70b" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:08.699066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.699043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4009dc87-5434-45df-9454-e6bc4d10b70b" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:08.699160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.699080    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q" (OuterVolumeSpecName: "kube-api-access-zk29q") pod "4009dc87-5434-45df-9454-e6bc4d10b70b" (UID: "4009dc87-5434-45df-9454-e6bc4d10b70b"). InnerVolumeSpecName "kube-api-access-zk29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:08.797899 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.797857    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4009dc87-5434-45df-9454-e6bc4d10b70b-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.797899 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.797894    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4009dc87-5434-45df-9454-e6bc4d10b70b-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.797899 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.797904    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.798095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.797913    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk29q\" (UniqueName: \"kubernetes.io/projected/4009dc87-5434-45df-9454-e6bc4d10b70b-kube-api-access-zk29q\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.798095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.797923    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4009dc87-5434-45df-9454-e6bc4d10b70b-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.994689 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.994597    2559 generic.go:356] "Generic (PLEG): container finished" podID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerID="61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed" exitCode=0
Jul 28 17:24:08.994689 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.994665    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"
Jul 28 17:24:08.994689 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.994676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerDied","Data":"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"}
Jul 28 17:24:08.995001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.994742    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf" event={"ID":"4009dc87-5434-45df-9454-e6bc4d10b70b","Type":"ContainerDied","Data":"21de5fd9a24b580d620ec9ff1e2556668857f38ec91f519a48a216ff45f799a5"}
Jul 28 17:24:08.995001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.994777    2559 scope.go:117] "RemoveContainer" containerID="86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"
Jul 28 17:24:08.995168 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.995155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:08.997379 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:08.997361    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:09.004171 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.004155    2559 scope.go:117] "RemoveContainer" containerID="61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"
Jul 28 17:24:09.011545 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.011529    2559 scope.go:117] "RemoveContainer" containerID="036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"
Jul 28 17:24:09.020167 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020149    2559 scope.go:117] "RemoveContainer" containerID="86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"
Jul 28 17:24:09.020449 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:09.020431    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49\": container with ID starting with 86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49 not found: ID does not exist" containerID="86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"
Jul 28 17:24:09.020517 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020463    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49"} err="failed to get container status \"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49\": rpc error: code = NotFound desc = could not find container \"86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49\": container with ID starting with 86ffb44a46dca35e6fb7c480661ae86547b485bf1cac17a673ca3ef18574af49 not found: ID does not exist"
Jul 28 17:24:09.020517 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020483    2559 scope.go:117] "RemoveContainer" containerID="61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"
Jul 28 17:24:09.020717 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:09.020699    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed\": container with ID starting with 61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed not found: ID does not exist" containerID="61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"
Jul 28 17:24:09.020772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020727    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed"} err="failed to get container status \"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed\": rpc error: code = NotFound desc = could not find container \"61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed\": container with ID starting with 61da50dfaa2e761e2b7fec1d0e3a555e0be4ff29d0fe12a3d451bb95801384ed not found: ID does not exist"
Jul 28 17:24:09.020772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020743    2559 scope.go:117] "RemoveContainer" containerID="036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"
Jul 28 17:24:09.020971 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:09.020954    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42\": container with ID starting with 036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42 not found: ID does not exist" containerID="036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"
Jul 28 17:24:09.021037 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.020983    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42"} err="failed to get container status \"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42\": rpc error: code = NotFound desc = could not find container \"036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42\": container with ID starting with 036cb7bf3b83627e2e86239377099ad34a76683a0277b799603b5ca20dbf9b42 not found: ID does not exist"
Jul 28 17:24:09.030447 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.030427    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:24:09.032916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.032897    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf"]
Jul 28 17:24:09.667828 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.667808    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:24:09.672338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.672317    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" path="/var/lib/kubelet/pods/4009dc87-5434-45df-9454-e6bc4d10b70b/volumes"
Jul 28 17:24:09.703947 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.703885    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcdmj\" (UniqueName: \"kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj\") pod \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") "
Jul 28 17:24:09.704068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.703960    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location\") pod \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") "
Jul 28 17:24:09.704068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704006    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config\") pod \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") "
Jul 28 17:24:09.704068 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls\") pod \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") "
Jul 28 17:24:09.704230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert\") pod \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\" (UID: \"2e18970d-c15c-4ec7-9807-e23fc4a172f0\") "
Jul 28 17:24:09.704332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704264    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e18970d-c15c-4ec7-9807-e23fc4a172f0" (UID: "2e18970d-c15c-4ec7-9807-e23fc4a172f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:09.704458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704434    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e18970d-c15c-4ec7-9807-e23fc4a172f0" (UID: "2e18970d-c15c-4ec7-9807-e23fc4a172f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:09.704531 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.704462    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e18970d-c15c-4ec7-9807-e23fc4a172f0" (UID: "2e18970d-c15c-4ec7-9807-e23fc4a172f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:09.705902 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.705853    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj" (OuterVolumeSpecName: "kube-api-access-vcdmj") pod "2e18970d-c15c-4ec7-9807-e23fc4a172f0" (UID: "2e18970d-c15c-4ec7-9807-e23fc4a172f0"). InnerVolumeSpecName "kube-api-access-vcdmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:09.706158 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.706140    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e18970d-c15c-4ec7-9807-e23fc4a172f0" (UID: "2e18970d-c15c-4ec7-9807-e23fc4a172f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:09.804764 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.804723    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdmj\" (UniqueName: \"kubernetes.io/projected/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-api-access-vcdmj\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.804764 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.804760    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.804764 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.804769    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.804764 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.804778    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e18970d-c15c-4ec7-9807-e23fc4a172f0-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:09.805018 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:09.804789    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e18970d-c15c-4ec7-9807-e23fc4a172f0-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:24:10.001577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.001490    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerID="bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498" exitCode=0
Jul 28 17:24:10.001577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.001535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerDied","Data":"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"}
Jul 28 17:24:10.001758 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.001669    2559 scope.go:117] "RemoveContainer" containerID="f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"
Jul 28 17:24:10.002027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.001597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b" event={"ID":"2e18970d-c15c-4ec7-9807-e23fc4a172f0","Type":"ContainerDied","Data":"1e76d710b73f45e2f1a0debe3b6a1047e5e5ab00844f4b3172b75a724be464f2"}
Jul 28 17:24:10.002027 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.002023    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"
Jul 28 17:24:10.017390 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.017373    2559 scope.go:117] "RemoveContainer" containerID="bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"
Jul 28 17:24:10.025607 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.025587    2559 scope.go:117] "RemoveContainer" containerID="6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"
Jul 28 17:24:10.030235 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.030216    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:24:10.033213 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.033192    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b"]
Jul 28 17:24:10.034446 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.034435    2559 scope.go:117] "RemoveContainer" containerID="f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"
Jul 28 17:24:10.034707 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:10.034682    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb\": container with ID starting with f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb not found: ID does not exist" containerID="f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"
Jul 28 17:24:10.034748 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.034707    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb"} err="failed to get container status \"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb\": rpc error: code = NotFound desc = could not find container \"f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb\": container with ID starting with f4b1482318104f1ef9209d17c93825a927d7e9f449e49f410a5e291bdc2bd9bb not found: ID does not exist"
Jul 28 17:24:10.034748 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.034723    2559 scope.go:117] "RemoveContainer" containerID="bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"
Jul 28 17:24:10.034989 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:10.034969    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498\": container with ID starting with bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498 not found: ID does not exist" containerID="bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"
Jul 28 17:24:10.035030 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.034997    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498"} err="failed to get container status \"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498\": rpc error: code = NotFound desc = could not find container \"bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498\": container with ID starting with bb81f5e87c34249379a715ed2151eeab89848e61325e93e8c735aee3a51a5498 not found: ID does not exist"
Jul 28 17:24:10.035030 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.035014    2559 scope.go:117] "RemoveContainer" containerID="6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"
Jul 28 17:24:10.035243 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:24:10.035224    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d\": container with ID starting with 6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d not found: ID does not exist" containerID="6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"
Jul 28 17:24:10.035299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:10.035251    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d"} err="failed to get container status \"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d\": rpc error: code = NotFound desc = could not find container \"6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d\": container with ID starting with 6800764e4ae9dcae4e13abb06468b8765d8a4a9e12b6cfaa7351893cdc2f505d not found: ID does not exist"
Jul 28 17:24:11.672640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:11.672609    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" path="/var/lib/kubelet/pods/2e18970d-c15c-4ec7-9807-e23fc4a172f0/volumes"
Jul 28 17:24:16.014190 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:16.014107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:24:21.558268 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:21.558242    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:24:21.559288 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:21.559267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:24:24.883365 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883678    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883700    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container"
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883705    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container"
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883717    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883723 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883723    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883733    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="storage-initializer"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883742    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="storage-initializer"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="storage-initializer"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883764    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="storage-initializer"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883780    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883833    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883841    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kube-rbac-proxy"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883848    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4009dc87-5434-45df-9454-e6bc4d10b70b" containerName="kserve-container"
Jul 28 17:24:24.883937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.883855    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e18970d-c15c-4ec7-9807-e23fc4a172f0" containerName="kserve-container"
Jul 28 17:24:24.888193 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.888177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:24.890329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.890308    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-b8148-predictor-serving-cert\""
Jul 28 17:24:24.890494 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.890476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:24:24.890548 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.890477    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-b8148-kube-rbac-proxy-sar-config\""
Jul 28 17:24:24.890730 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.890714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:24:24.898948 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.898926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:24:24.927667 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.927643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:24.927814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.927676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:24.927814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.927699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mrk\" (UniqueName: \"kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:24.927814 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.927798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:24.927942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:24.927832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.028613 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.028582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.028808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.028618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.028808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.028643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mrk\" (UniqueName: \"kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.028808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.028677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.028808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.028699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.029207 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.029182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.029381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.029361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.029445 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.029360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.031032 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.031016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.039297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.039275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mrk\" (UniqueName: \"kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk\") pod \"isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.199230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.199148    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:25.338274 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:25.338245    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:24:25.340584 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:24:25.340559    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67226f7_f925_43a1_9c6d_7423b322dca6.slice/crio-98ca266eb7aeaedf6623e6068867966a484f3d23551a23e2657c3cb6fe6190da WatchSource:0}: Error finding container 98ca266eb7aeaedf6623e6068867966a484f3d23551a23e2657c3cb6fe6190da: Status 404 returned error can't find the container with id 98ca266eb7aeaedf6623e6068867966a484f3d23551a23e2657c3cb6fe6190da
Jul 28 17:24:26.076495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:26.076463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerStarted","Data":"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"}
Jul 28 17:24:26.076831 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:26.076521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerStarted","Data":"98ca266eb7aeaedf6623e6068867966a484f3d23551a23e2657c3cb6fe6190da"}
Jul 28 17:24:30.098188 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:30.098157    2559 generic.go:356] "Generic (PLEG): container finished" podID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerID="042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144" exitCode=0
Jul 28 17:24:30.098557 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:30.098230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerDied","Data":"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"}
Jul 28 17:24:31.105297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.105265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerStarted","Data":"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"}
Jul 28 17:24:31.105717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.105329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerStarted","Data":"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"}
Jul 28 17:24:31.105717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.105355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerStarted","Data":"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"}
Jul 28 17:24:31.105717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.105626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:31.105717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.105681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:31.107061 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.107039    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:31.131383 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:31.131335    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podStartSLOduration=3.267306723 podStartE2EDuration="7.131320666s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864013943s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:31.129372748 +0000 UTC m=+910.011779883" watchObservedRunningTime="2026-07-28 17:24:31.131320666 +0000 UTC m=+910.013727781"
Jul 28 17:24:32.110155 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:32.110116    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:32.110155 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:32.110152    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:32.111358 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:32.111331    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:33.114157 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:33.114112    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:33.114636 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:33.114519    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:38.118832 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:38.118806    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:24:38.119626 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:38.119333    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:38.119731 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:38.119647    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:48.119330 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:48.119272    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:48.121495 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:48.119703    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:58.119354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:58.119313    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:24:58.119953 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:24:58.119832    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:08.119813 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:08.119755    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:25:08.120472 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:08.120317    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:18.119951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:18.119908    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:25:18.120548 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:18.120424    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:28.120189 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:28.120143    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:25:28.120688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:28.120625    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:38.119324 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:38.119279    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:25:38.119787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:38.119726    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:44.668453 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:44.668383    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:25:44.668816 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:44.668634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:25:59.903129 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:25:59.903100    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-b8148-predictor-86f78f7649-2xqrx_6f282bca-8e26-4e97-b1a4-0f193ac3ec56/kserve-container/0.log"
Jul 28 17:26:00.157530 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.157452    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:26:00.157761 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.157712    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" containerID="cri-o://3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556" gracePeriod=30
Jul 28 17:26:00.157860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.157724    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" containerID="cri-o://d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e" gracePeriod=30
Jul 28 17:26:00.157860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.157736    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" containerID="cri-o://f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02" gracePeriod=30
Jul 28 17:26:00.242310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.242275    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:26:00.253995 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.253964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.256568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.256545    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e2d9f-predictor-serving-cert\""
Jul 28 17:26:00.256841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.256557    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-e2d9f-kube-rbac-proxy-sar-config\""
Jul 28 17:26:00.262179 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.262052    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:26:00.282558 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.282532    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:26:00.282768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.282744    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kserve-container" containerID="cri-o://2ba26293fe79bc5817cc54f0eb6d5e05e118ba3d788895b1e0b25105b70fbccc" gracePeriod=30
Jul 28 17:26:00.282899 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.282799    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kube-rbac-proxy" containerID="cri-o://766b9fc522278c01d5dd1fcbe5ea3f05a4bd8e06745b2b65bcb6f43a966d2d64" gracePeriod=30
Jul 28 17:26:00.405508 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.405484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.405614 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.405530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.405651 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.405644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbcx\" (UniqueName: \"kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.405686 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.405675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.405724 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.405701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506370 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbcx\" (UniqueName: \"kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.506932 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:00.506668    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-serving-cert: secret "isvc-sklearn-scale-raw-e2d9f-predictor-serving-cert" not found
Jul 28 17:26:00.506932 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:00.506769    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls podName:ce11840e-1649-47b0-bfef-f9b1774706f1 nodeName:}" failed. No retries permitted until 2026-07-28 17:26:01.006744087 +0000 UTC m=+999.889151190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls") pod "isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1") : secret "isvc-sklearn-scale-raw-e2d9f-predictor-serving-cert" not found
Jul 28 17:26:00.506932 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.506918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.507345 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.507311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.507439 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.507422    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.515434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.515409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbcx\" (UniqueName: \"kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:00.531232 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.531203    2559 generic.go:356] "Generic (PLEG): container finished" podID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerID="f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02" exitCode=2
Jul 28 17:26:00.531347 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.531289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerDied","Data":"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"}
Jul 28 17:26:00.532666 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532645    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerID="766b9fc522278c01d5dd1fcbe5ea3f05a4bd8e06745b2b65bcb6f43a966d2d64" exitCode=2
Jul 28 17:26:00.532666 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532663    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerID="2ba26293fe79bc5817cc54f0eb6d5e05e118ba3d788895b1e0b25105b70fbccc" exitCode=2
Jul 28 17:26:00.532804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerDied","Data":"766b9fc522278c01d5dd1fcbe5ea3f05a4bd8e06745b2b65bcb6f43a966d2d64"}
Jul 28 17:26:00.532804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerDied","Data":"2ba26293fe79bc5817cc54f0eb6d5e05e118ba3d788895b1e0b25105b70fbccc"}
Jul 28 17:26:00.532804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx" event={"ID":"6f282bca-8e26-4e97-b1a4-0f193ac3ec56","Type":"ContainerDied","Data":"8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd"}
Jul 28 17:26:00.532804 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.532750    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bbc6296ef0e4794f95d7c04767c375a83cdf15dc7205fffb5c662aa3b0c0abd"
Jul 28 17:26:00.534654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.534640    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:26:00.708153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.708081    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") pod \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") "
Jul 28 17:26:00.708153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.708122    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config\") pod \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") "
Jul 28 17:26:00.708153 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.708149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnnq9\" (UniqueName: \"kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9\") pod \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\" (UID: \"6f282bca-8e26-4e97-b1a4-0f193ac3ec56\") "
Jul 28 17:26:00.708474 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.708444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f282bca-8e26-4e97-b1a4-0f193ac3ec56" (UID: "6f282bca-8e26-4e97-b1a4-0f193ac3ec56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:00.710123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.710103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f282bca-8e26-4e97-b1a4-0f193ac3ec56" (UID: "6f282bca-8e26-4e97-b1a4-0f193ac3ec56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:00.710187 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.710129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9" (OuterVolumeSpecName: "kube-api-access-gnnq9") pod "6f282bca-8e26-4e97-b1a4-0f193ac3ec56" (UID: "6f282bca-8e26-4e97-b1a4-0f193ac3ec56"). InnerVolumeSpecName "kube-api-access-gnnq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:00.809258 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.809228    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:00.809258 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.809251    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:00.809258 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:00.809261    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnnq9\" (UniqueName: \"kubernetes.io/projected/6f282bca-8e26-4e97-b1a4-0f193ac3ec56-kube-api-access-gnnq9\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:01.011750 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.011663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:01.014106 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.014084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") pod \"isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:01.168137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.168108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:01.309348 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.309323    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:26:01.311262 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:26:01.311234    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce11840e_1649_47b0_bfef_f9b1774706f1.slice/crio-e7a3063086c0f868fc5bfba3f518cca75b8bf9db80149b993b855a04931272fb WatchSource:0}: Error finding container e7a3063086c0f868fc5bfba3f518cca75b8bf9db80149b993b855a04931272fb: Status 404 returned error can't find the container with id e7a3063086c0f868fc5bfba3f518cca75b8bf9db80149b993b855a04931272fb
Jul 28 17:26:01.540151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.540125    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"
Jul 28 17:26:01.540151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.540123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerStarted","Data":"fd3e571179c81ca2c608db21b9d4aab5089c991f3bf2308f072e73b9d8288d55"}
Jul 28 17:26:01.540359 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.540197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerStarted","Data":"e7a3063086c0f868fc5bfba3f518cca75b8bf9db80149b993b855a04931272fb"}
Jul 28 17:26:01.575690 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.575659    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:26:01.579558 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.579534    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx"]
Jul 28 17:26:01.673736 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:01.673703    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" path="/var/lib/kubelet/pods/6f282bca-8e26-4e97-b1a4-0f193ac3ec56/volumes"
Jul 28 17:26:03.114917 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:03.114848    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:04.667891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:04.667833    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:26:04.668277 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:04.668172    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:05.562706 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:05.562677    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerID="fd3e571179c81ca2c608db21b9d4aab5089c991f3bf2308f072e73b9d8288d55" exitCode=0
Jul 28 17:26:05.562848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:05.562751    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerDied","Data":"fd3e571179c81ca2c608db21b9d4aab5089c991f3bf2308f072e73b9d8288d55"}
Jul 28 17:26:05.565317 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:05.565293    2559 generic.go:356] "Generic (PLEG): container finished" podID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerID="3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556" exitCode=0
Jul 28 17:26:05.565442 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:05.565316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerDied","Data":"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"}
Jul 28 17:26:06.571490 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:06.571456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerStarted","Data":"d1dba7901440ee90eff86c768b8094680dce09d9897ac4e83cf281a5b6987e2d"}
Jul 28 17:26:06.571898 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:06.571520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerStarted","Data":"e77e5265bf0d3d48f946a6aa3ec35509af78e1fa418df4ae209733e5368c8ccb"}
Jul 28 17:26:06.571898 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:06.571677    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:06.593092 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:06.593048    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podStartSLOduration=2.738275561 podStartE2EDuration="6.593034878s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.854759317s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:06.591060894 +0000 UTC m=+1005.473468029" watchObservedRunningTime="2026-07-28 17:26:06.593034878 +0000 UTC m=+1005.475441993"
Jul 28 17:26:07.576756 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:07.576726    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:07.577969 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:07.577945    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:08.115230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:08.115193    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:08.581423 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:08.581381    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:13.114986 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:13.114939    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:13.115413 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:13.115072    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:26:13.586638 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:13.586603    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:26:13.587246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:13.587209    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:14.667977 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:14.667937    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:26:14.668370 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:14.668256    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:18.114903 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:18.114845    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:23.114434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:23.114390    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:23.587793 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:23.587759    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:24.667834 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:24.667781    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:26:24.668373 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:24.667929    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:26:24.668373 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:24.668114    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:24.668373 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:24.668254    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:26:28.114927 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:28.114863    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:26:30.313709 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.313683    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:26:30.337668 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.337639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4mrk\" (UniqueName: \"kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk\") pod \"c67226f7-f925-43a1-9c6d-7423b322dca6\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") "
Jul 28 17:26:30.337848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.337695    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location\") pod \"c67226f7-f925-43a1-9c6d-7423b322dca6\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") "
Jul 28 17:26:30.337848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.337718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert\") pod \"c67226f7-f925-43a1-9c6d-7423b322dca6\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") "
Jul 28 17:26:30.337848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.337774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls\") pod \"c67226f7-f925-43a1-9c6d-7423b322dca6\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") "
Jul 28 17:26:30.337848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.337800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config\") pod \"c67226f7-f925-43a1-9c6d-7423b322dca6\" (UID: \"c67226f7-f925-43a1-9c6d-7423b322dca6\") "
Jul 28 17:26:30.338254 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.338212    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c67226f7-f925-43a1-9c6d-7423b322dca6" (UID: "c67226f7-f925-43a1-9c6d-7423b322dca6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:30.338382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.338264    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c67226f7-f925-43a1-9c6d-7423b322dca6" (UID: "c67226f7-f925-43a1-9c6d-7423b322dca6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:30.338602 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.338575    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c67226f7-f925-43a1-9c6d-7423b322dca6" (UID: "c67226f7-f925-43a1-9c6d-7423b322dca6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:30.339962 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.339928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk" (OuterVolumeSpecName: "kube-api-access-t4mrk") pod "c67226f7-f925-43a1-9c6d-7423b322dca6" (UID: "c67226f7-f925-43a1-9c6d-7423b322dca6"). InnerVolumeSpecName "kube-api-access-t4mrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:30.340230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.340210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c67226f7-f925-43a1-9c6d-7423b322dca6" (UID: "c67226f7-f925-43a1-9c6d-7423b322dca6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:30.438892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.438786    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mrk\" (UniqueName: \"kubernetes.io/projected/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-api-access-t4mrk\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.438892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.438814    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c67226f7-f925-43a1-9c6d-7423b322dca6-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.438892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.438824    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.438892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.438834    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c67226f7-f925-43a1-9c6d-7423b322dca6-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.438892 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.438842    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c67226f7-f925-43a1-9c6d-7423b322dca6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.689622 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.689544    2559 generic.go:356] "Generic (PLEG): container finished" podID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerID="d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e" exitCode=0
Jul 28 17:26:30.689622 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.689600    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"
Jul 28 17:26:30.689893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.689602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerDied","Data":"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"}
Jul 28 17:26:30.689893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.689738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7" event={"ID":"c67226f7-f925-43a1-9c6d-7423b322dca6","Type":"ContainerDied","Data":"98ca266eb7aeaedf6623e6068867966a484f3d23551a23e2657c3cb6fe6190da"}
Jul 28 17:26:30.689893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.689781    2559 scope.go:117] "RemoveContainer" containerID="d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"
Jul 28 17:26:30.698817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.698801    2559 scope.go:117] "RemoveContainer" containerID="f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"
Jul 28 17:26:30.708230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.708211    2559 scope.go:117] "RemoveContainer" containerID="3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"
Jul 28 17:26:30.713435 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.713411    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:26:30.716472 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.716452    2559 scope.go:117] "RemoveContainer" containerID="042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"
Jul 28 17:26:30.717750 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.717733    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7"]
Jul 28 17:26:30.724382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.724368    2559 scope.go:117] "RemoveContainer" containerID="d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"
Jul 28 17:26:30.724622 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:30.724605    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e\": container with ID starting with d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e not found: ID does not exist" containerID="d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"
Jul 28 17:26:30.724673 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.724634    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e"} err="failed to get container status \"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e\": rpc error: code = NotFound desc = could not find container \"d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e\": container with ID starting with d6c3afb06c200a26bd08ea0758dea26b661792d344001e6896587ee9ef958e0e not found: ID does not exist"
Jul 28 17:26:30.724673 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.724652    2559 scope.go:117] "RemoveContainer" containerID="f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"
Jul 28 17:26:30.724863 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:30.724849    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02\": container with ID starting with f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02 not found: ID does not exist" containerID="f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"
Jul 28 17:26:30.724924 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.724865    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02"} err="failed to get container status \"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02\": rpc error: code = NotFound desc = could not find container \"f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02\": container with ID starting with f205164cb409d1de6a20196e413dca824b0a0caa8cd1d4f0b9a1ad64ff2b2e02 not found: ID does not exist"
Jul 28 17:26:30.724924 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.724894    2559 scope.go:117] "RemoveContainer" containerID="3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"
Jul 28 17:26:30.725105 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:30.725084    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556\": container with ID starting with 3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556 not found: ID does not exist" containerID="3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"
Jul 28 17:26:30.725144 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.725108    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556"} err="failed to get container status \"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556\": rpc error: code = NotFound desc = could not find container \"3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556\": container with ID starting with 3d1efc2e6126a5e1cbf1c587291dfc76ece768a09a2408ed4bdfe520ca159556 not found: ID does not exist"
Jul 28 17:26:30.725144 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.725125    2559 scope.go:117] "RemoveContainer" containerID="042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"
Jul 28 17:26:30.725353 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:26:30.725333    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144\": container with ID starting with 042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144 not found: ID does not exist" containerID="042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"
Jul 28 17:26:30.725441 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:30.725357    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144"} err="failed to get container status \"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144\": rpc error: code = NotFound desc = could not find container \"042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144\": container with ID starting with 042b08119eb513ee1c114a550127341a7c76017035b1c7b51f5f385d05111144 not found: ID does not exist"
Jul 28 17:26:31.672604 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:31.672571    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" path="/var/lib/kubelet/pods/c67226f7-f925-43a1-9c6d-7423b322dca6/volumes"
Jul 28 17:26:33.587443 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:33.587408    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:43.587388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:43.587352    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:26:53.588107 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:26:53.588066    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:03.587448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:03.587408    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:13.587735 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:13.587644    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:23.588010 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:23.587975    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:33.588041 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:33.588004    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:34.667726 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:34.667681    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:44.667657 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:44.667613    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:27:54.668646 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:27:54.668601    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:04.667769 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:04.667721    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:14.667693 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:14.667649    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:24.667778 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:24.667731    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:34.668256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:34.668205    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:44.668829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:44.668743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:28:50.572065 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:50.572033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:28:50.572474 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:50.572276    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" containerID="cri-o://e77e5265bf0d3d48f946a6aa3ec35509af78e1fa418df4ae209733e5368c8ccb" gracePeriod=30
Jul 28 17:28:50.572474 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:50.572321    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy" containerID="cri-o://d1dba7901440ee90eff86c768b8094680dce09d9897ac4e83cf281a5b6987e2d" gracePeriod=30
Jul 28 17:28:51.334521 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:51.334490    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerID="d1dba7901440ee90eff86c768b8094680dce09d9897ac4e83cf281a5b6987e2d" exitCode=2
Jul 28 17:28:51.334696 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:51.334550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerDied","Data":"d1dba7901440ee90eff86c768b8094680dce09d9897ac4e83cf281a5b6987e2d"}
Jul 28 17:28:53.582230 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:53.582193    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 28 17:28:54.668300 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:54.668261    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:28:55.698173 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698142    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698475    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kube-rbac-proxy"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698486    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kube-rbac-proxy"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698496    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698514    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698520    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698527    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="storage-initializer"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698533    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="storage-initializer"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698541    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container"
Jul 28 17:28:55.698541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698546    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698555    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kserve-container"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698560    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kserve-container"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698630    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kserve-container"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698637    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="agent"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kserve-container"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698649    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67226f7-f925-43a1-9c6d-7423b322dca6" containerName="kube-rbac-proxy"
Jul 28 17:28:55.698861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.698654    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f282bca-8e26-4e97-b1a4-0f193ac3ec56" containerName="kube-rbac-proxy"
Jul 28 17:28:55.701798 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.701783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.704539 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.704516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:28:55.705505 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.705481    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:28:55.705903 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.705862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:28:55.709108 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.709091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.711850 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.711829    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:28:55.720643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.720625    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:28:55.722741 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.722722    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:28:55.811091 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.811240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.811240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.811240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbk72\" (UniqueName: \"kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.811240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.811374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.811374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spppc\" (UniqueName: \"kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.811374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.811374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.811374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.811364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.911986 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.911950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.911986 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.911989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbk72\" (UniqueName: \"kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912481 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912481 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spppc\" (UniqueName: \"kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912481 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.912916 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.912815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.912916 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:28:55.912898    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:28:55.913204 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:28:55.912979    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls podName:7ebc4a67-4112-4b34-be98-97993e3b6619 nodeName:}" failed. No retries permitted until 2026-07-28 17:28:56.412957745 +0000 UTC m=+1175.295364841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls") pod "isvc-canary-create-v2-predictor-5b976d889b-7pg8g" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:28:55.913204 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.913100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:55.914552 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.914533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.922715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.922687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spppc\" (UniqueName: \"kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc\") pod \"isvc-canary-create-predictor-594f5f9bd5-krwrj\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:55.923123 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:55.923102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbk72\" (UniqueName: \"kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:56.012255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.012187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:28:56.154115 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.154070    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:28:56.156799 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:28:56.156772    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83cba32_650c_4bc3_8c0f_f509fa0fdff4.slice/crio-2913da79e5c1bc09d264f376bac10f19235e598ede7e567f55f9d290e8b8e283 WatchSource:0}: Error finding container 2913da79e5c1bc09d264f376bac10f19235e598ede7e567f55f9d290e8b8e283: Status 404 returned error can't find the container with id 2913da79e5c1bc09d264f376bac10f19235e598ede7e567f55f9d290e8b8e283
Jul 28 17:28:56.359690 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.359656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerStarted","Data":"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"}
Jul 28 17:28:56.359852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.359727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerStarted","Data":"2913da79e5c1bc09d264f376bac10f19235e598ede7e567f55f9d290e8b8e283"}
Jul 28 17:28:56.417724 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.417687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:56.420030 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.420003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-7pg8g\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:56.620747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.620665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:28:56.763507 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:56.763360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:28:56.766826 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:28:56.766800    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebc4a67_4112_4b34_be98_97993e3b6619.slice/crio-5148e1fb3476bacd4e85d5e00ba9d2e34272d27733fa609a3be9abcc278a3bf8 WatchSource:0}: Error finding container 5148e1fb3476bacd4e85d5e00ba9d2e34272d27733fa609a3be9abcc278a3bf8: Status 404 returned error can't find the container with id 5148e1fb3476bacd4e85d5e00ba9d2e34272d27733fa609a3be9abcc278a3bf8
Jul 28 17:28:57.367328 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:57.367292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerStarted","Data":"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"}
Jul 28 17:28:57.367551 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:57.367361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerStarted","Data":"5148e1fb3476bacd4e85d5e00ba9d2e34272d27733fa609a3be9abcc278a3bf8"}
Jul 28 17:28:58.582165 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:28:58.582128    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 28 17:29:00.389331 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.389301    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerID="e77e5265bf0d3d48f946a6aa3ec35509af78e1fa418df4ae209733e5368c8ccb" exitCode=0
Jul 28 17:29:00.389715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.389375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerDied","Data":"e77e5265bf0d3d48f946a6aa3ec35509af78e1fa418df4ae209733e5368c8ccb"}
Jul 28 17:29:00.390845 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.390823    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerID="baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32" exitCode=0
Jul 28 17:29:00.390987 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.390857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerDied","Data":"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"}
Jul 28 17:29:00.529690 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.529667    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:29:00.654099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config\") pod \"ce11840e-1649-47b0-bfef-f9b1774706f1\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") "
Jul 28 17:29:00.654257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nbcx\" (UniqueName: \"kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx\") pod \"ce11840e-1649-47b0-bfef-f9b1774706f1\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") "
Jul 28 17:29:00.654257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert\") pod \"ce11840e-1649-47b0-bfef-f9b1774706f1\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") "
Jul 28 17:29:00.654257 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654251    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") pod \"ce11840e-1649-47b0-bfef-f9b1774706f1\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") "
Jul 28 17:29:00.654419 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654289    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location\") pod \"ce11840e-1649-47b0-bfef-f9b1774706f1\" (UID: \"ce11840e-1649-47b0-bfef-f9b1774706f1\") "
Jul 28 17:29:00.654465 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654424    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce11840e-1649-47b0-bfef-f9b1774706f1" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:00.654563 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654534    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:29:00.654621 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce11840e-1649-47b0-bfef-f9b1774706f1" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:00.654667 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.654606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce11840e-1649-47b0-bfef-f9b1774706f1" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:29:00.656113 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.656092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx" (OuterVolumeSpecName: "kube-api-access-4nbcx") pod "ce11840e-1649-47b0-bfef-f9b1774706f1" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1"). InnerVolumeSpecName "kube-api-access-4nbcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:29:00.656206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.656171    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce11840e-1649-47b0-bfef-f9b1774706f1" (UID: "ce11840e-1649-47b0-bfef-f9b1774706f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:29:00.755567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.755490    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce11840e-1649-47b0-bfef-f9b1774706f1-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:29:00.755567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.755517    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce11840e-1649-47b0-bfef-f9b1774706f1-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:29:00.755567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.755527    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbcx\" (UniqueName: \"kubernetes.io/projected/ce11840e-1649-47b0-bfef-f9b1774706f1-kube-api-access-4nbcx\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:29:00.755567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:00.755536    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce11840e-1649-47b0-bfef-f9b1774706f1-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:29:01.397388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.397355    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"
Jul 28 17:29:01.397388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.397369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4" event={"ID":"ce11840e-1649-47b0-bfef-f9b1774706f1","Type":"ContainerDied","Data":"e7a3063086c0f868fc5bfba3f518cca75b8bf9db80149b993b855a04931272fb"}
Jul 28 17:29:01.397926 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.397453    2559 scope.go:117] "RemoveContainer" containerID="d1dba7901440ee90eff86c768b8094680dce09d9897ac4e83cf281a5b6987e2d"
Jul 28 17:29:01.399745 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.399725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerStarted","Data":"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"}
Jul 28 17:29:01.399893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.399782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerStarted","Data":"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"}
Jul 28 17:29:01.400047 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.400026    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:29:01.405200 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.405181    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerID="d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30" exitCode=0
Jul 28 17:29:01.405293 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.405228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerDied","Data":"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"}
Jul 28 17:29:01.408953 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.408939    2559 scope.go:117] "RemoveContainer" containerID="e77e5265bf0d3d48f946a6aa3ec35509af78e1fa418df4ae209733e5368c8ccb"
Jul 28 17:29:01.416194 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.416180    2559 scope.go:117] "RemoveContainer" containerID="fd3e571179c81ca2c608db21b9d4aab5089c991f3bf2308f072e73b9d8288d55"
Jul 28 17:29:01.494524 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.494484    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podStartSLOduration=2.544138908 podStartE2EDuration="6.494468062s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.950329154s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:01.457491866 +0000 UTC m=+1180.339898992" watchObservedRunningTime="2026-07-28 17:29:01.494468062 +0000 UTC m=+1180.376875179"
Jul 28 17:29:01.517178 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.517155    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:29:01.525742 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.525716    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4"]
Jul 28 17:29:01.673012 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:01.672985    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" path="/var/lib/kubelet/pods/ce11840e-1649-47b0-bfef-f9b1774706f1/volumes"
Jul 28 17:29:02.413246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.413206    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerStarted","Data":"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"}
Jul 28 17:29:02.413698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.413299    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerStarted","Data":"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"}
Jul 28 17:29:02.413698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.413520    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:29:02.413830 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.413787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:29:02.413909 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.413837    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:29:02.414721 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.414700    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:02.414810 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.414783    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:02.436831 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:02.436787    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podStartSLOduration=3.6537379039999998 podStartE2EDuration="7.436774003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783036099s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:02.435292711 +0000 UTC m=+1181.317699832" watchObservedRunningTime="2026-07-28 17:29:02.436774003 +0000 UTC m=+1181.319181119"
Jul 28 17:29:03.419650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:03.418603    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:03.419650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:03.418921    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:08.424076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:08.424045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:29:08.424555 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:08.424147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:29:08.424696 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:08.424662    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:08.424825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:08.424703    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:18.424931 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:18.424890    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:18.425467 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:18.424890    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:21.594384 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:21.594350    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:29:21.596697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:21.596677    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:29:28.425076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:28.425035    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:28.425076 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:28.425045    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:38.424654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:38.424610    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:38.425176 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:38.424610    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:48.425098 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:48.425054    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:29:48.427412 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:48.425114    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:58.424881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:58.424813    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:29:58.424881 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:29:58.424835    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:30:08.424664 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:08.424617    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:30:08.425048 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:08.424625    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:30:18.425067 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:18.424981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:30:18.425431 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:18.425070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:30:21.643674 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:21.643649    2559 scope.go:117] "RemoveContainer" containerID="766b9fc522278c01d5dd1fcbe5ea3f05a4bd8e06745b2b65bcb6f43a966d2d64"
Jul 28 17:30:21.651958 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:21.651914    2559 scope.go:117] "RemoveContainer" containerID="2ba26293fe79bc5817cc54f0eb6d5e05e118ba3d788895b1e0b25105b70fbccc"
Jul 28 17:30:25.774938 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.774908    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:30:25.776151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.775521    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kube-rbac-proxy" containerID="cri-o://a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85" gracePeriod=30
Jul 28 17:30:25.776151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.775672    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" containerID="cri-o://301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532" gracePeriod=30
Jul 28 17:30:25.778018 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.777996    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:30:25.778213 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.778195    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" containerID="cri-o://07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1" gracePeriod=30
Jul 28 17:30:25.778345 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:25.778298    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kube-rbac-proxy" containerID="cri-o://f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e" gracePeriod=30
Jul 28 17:30:26.831042 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:26.831006    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerID="f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e" exitCode=2
Jul 28 17:30:26.831491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:26.831083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerDied","Data":"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"}
Jul 28 17:30:26.833058 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:26.833039    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerID="a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85" exitCode=2
Jul 28 17:30:26.833176 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:26.833065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerDied","Data":"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"}
Jul 28 17:30:28.418503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:28.418461    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 28 17:30:28.418858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:28.418464    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 28 17:30:28.425214 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:28.425186    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:30:28.425338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:28.425188    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:30:30.432265 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.432242    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:30:30.501363 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501289    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls\") pod \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") "
Jul 28 17:30:30.501503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config\") pod \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") "
Jul 28 17:30:30.501503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501405    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert\") pod \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") "
Jul 28 17:30:30.501503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501436    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spppc\" (UniqueName: \"kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc\") pod \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") "
Jul 28 17:30:30.501503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location\") pod \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\" (UID: \"d83cba32-650c-4bc3-8c0f-f509fa0fdff4\") "
Jul 28 17:30:30.501852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d83cba32-650c-4bc3-8c0f-f509fa0fdff4" (UID: "d83cba32-650c-4bc3-8c0f-f509fa0fdff4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:30.501966 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501829    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d83cba32-650c-4bc3-8c0f-f509fa0fdff4" (UID: "d83cba32-650c-4bc3-8c0f-f509fa0fdff4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:30.501966 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.501884    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d83cba32-650c-4bc3-8c0f-f509fa0fdff4" (UID: "d83cba32-650c-4bc3-8c0f-f509fa0fdff4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:30.503854 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.503829    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc" (OuterVolumeSpecName: "kube-api-access-spppc") pod "d83cba32-650c-4bc3-8c0f-f509fa0fdff4" (UID: "d83cba32-650c-4bc3-8c0f-f509fa0fdff4"). InnerVolumeSpecName "kube-api-access-spppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:30.504063 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.503834    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d83cba32-650c-4bc3-8c0f-f509fa0fdff4" (UID: "d83cba32-650c-4bc3-8c0f-f509fa0fdff4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:30.532600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.532575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:30:30.602314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602286    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location\") pod \"7ebc4a67-4112-4b34-be98-97993e3b6619\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") "
Jul 28 17:30:30.602443 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") pod \"7ebc4a67-4112-4b34-be98-97993e3b6619\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") "
Jul 28 17:30:30.602443 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config\") pod \"7ebc4a67-4112-4b34-be98-97993e3b6619\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") "
Jul 28 17:30:30.602556 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert\") pod \"7ebc4a67-4112-4b34-be98-97993e3b6619\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") "
Jul 28 17:30:30.602603 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602573    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ebc4a67-4112-4b34-be98-97993e3b6619" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:30.602674 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbk72\" (UniqueName: \"kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72\") pod \"7ebc4a67-4112-4b34-be98-97993e3b6619\" (UID: \"7ebc4a67-4112-4b34-be98-97993e3b6619\") "
Jul 28 17:30:30.602797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602763    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ebc4a67-4112-4b34-be98-97993e3b6619" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:30.602936 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ebc4a67-4112-4b34-be98-97993e3b6619" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:30.602936 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602904    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebc4a67-4112-4b34-be98-97993e3b6619-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.602936 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602923    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.602936 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602933    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.603121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602943    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.603121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602952    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spppc\" (UniqueName: \"kubernetes.io/projected/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kube-api-access-spppc\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.603121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602960    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.603121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.602969    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d83cba32-650c-4bc3-8c0f-f509fa0fdff4-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.604381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.604366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ebc4a67-4112-4b34-be98-97993e3b6619" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:30.604515 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.604493    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72" (OuterVolumeSpecName: "kube-api-access-qbk72") pod "7ebc4a67-4112-4b34-be98-97993e3b6619" (UID: "7ebc4a67-4112-4b34-be98-97993e3b6619"). InnerVolumeSpecName "kube-api-access-qbk72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:30.704332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.704300    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbk72\" (UniqueName: \"kubernetes.io/projected/7ebc4a67-4112-4b34-be98-97993e3b6619-kube-api-access-qbk72\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.704332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.704328    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebc4a67-4112-4b34-be98-97993e3b6619-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.704490 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.704344    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc4a67-4112-4b34-be98-97993e3b6619-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:30:30.852956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.852804    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerID="07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1" exitCode=0
Jul 28 17:30:30.852956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.852911    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"
Jul 28 17:30:30.852956 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.852921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerDied","Data":"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"}
Jul 28 17:30:30.853310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.852990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj" event={"ID":"d83cba32-650c-4bc3-8c0f-f509fa0fdff4","Type":"ContainerDied","Data":"2913da79e5c1bc09d264f376bac10f19235e598ede7e567f55f9d290e8b8e283"}
Jul 28 17:30:30.853310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.853029    2559 scope.go:117] "RemoveContainer" containerID="f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"
Jul 28 17:30:30.855223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.855197    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerID="301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532" exitCode=0
Jul 28 17:30:30.855353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.855275    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"
Jul 28 17:30:30.855353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.855318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerDied","Data":"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"}
Jul 28 17:30:30.855470 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.855368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g" event={"ID":"7ebc4a67-4112-4b34-be98-97993e3b6619","Type":"ContainerDied","Data":"5148e1fb3476bacd4e85d5e00ba9d2e34272d27733fa609a3be9abcc278a3bf8"}
Jul 28 17:30:30.862404 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.862387    2559 scope.go:117] "RemoveContainer" containerID="07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"
Jul 28 17:30:30.872081 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.872061    2559 scope.go:117] "RemoveContainer" containerID="baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"
Jul 28 17:30:30.879717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.879699    2559 scope.go:117] "RemoveContainer" containerID="f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"
Jul 28 17:30:30.879955 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.879929    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e\": container with ID starting with f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e not found: ID does not exist" containerID="f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"
Jul 28 17:30:30.880016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.879956    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e"} err="failed to get container status \"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e\": rpc error: code = NotFound desc = could not find container \"f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e\": container with ID starting with f0dee6cbd62d5081c3ae493638e0011aacd923a625309b416bb9b07ff83f742e not found: ID does not exist"
Jul 28 17:30:30.880016 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.879975    2559 scope.go:117] "RemoveContainer" containerID="07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"
Jul 28 17:30:30.880180 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.880166    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1\": container with ID starting with 07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1 not found: ID does not exist" containerID="07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"
Jul 28 17:30:30.880221 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.880185    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1"} err="failed to get container status \"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1\": rpc error: code = NotFound desc = could not find container \"07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1\": container with ID starting with 07e1590d07dbf4f46e7d3d7fde750994a03e16e48ebd71c14c9e4378486a35f1 not found: ID does not exist"
Jul 28 17:30:30.880221 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.880199    2559 scope.go:117] "RemoveContainer" containerID="baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"
Jul 28 17:30:30.880397 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.880370    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32\": container with ID starting with baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32 not found: ID does not exist" containerID="baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"
Jul 28 17:30:30.880462 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.880406    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32"} err="failed to get container status \"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32\": rpc error: code = NotFound desc = could not find container \"baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32\": container with ID starting with baaace37151e0fcf0a27ead01091228a260daa9bb98ac0f4de13218f3fa36b32 not found: ID does not exist"
Jul 28 17:30:30.880462 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.880429    2559 scope.go:117] "RemoveContainer" containerID="a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"
Jul 28 17:30:30.885509 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.885484    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:30:30.888210 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.888197    2559 scope.go:117] "RemoveContainer" containerID="301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"
Jul 28 17:30:30.894609 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.894580    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g"]
Jul 28 17:30:30.896213 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.896194    2559 scope.go:117] "RemoveContainer" containerID="d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"
Jul 28 17:30:30.910243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910218    2559 scope.go:117] "RemoveContainer" containerID="a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"
Jul 28 17:30:30.910520 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910499    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:30:30.910583 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.910516    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85\": container with ID starting with a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85 not found: ID does not exist" containerID="a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"
Jul 28 17:30:30.910583 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910545    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85"} err="failed to get container status \"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85\": rpc error: code = NotFound desc = could not find container \"a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85\": container with ID starting with a9921c53aad53d27701b120998d5d55703517b7dfc62bd33b429eb6c0107fa85 not found: ID does not exist"
Jul 28 17:30:30.910583 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910561    2559 scope.go:117] "RemoveContainer" containerID="301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"
Jul 28 17:30:30.910790 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.910774    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532\": container with ID starting with 301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532 not found: ID does not exist" containerID="301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"
Jul 28 17:30:30.910836 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910794    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532"} err="failed to get container status \"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532\": rpc error: code = NotFound desc = could not find container \"301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532\": container with ID starting with 301c16e4115f4926a0c0472df66bd163bcad894ea191ceef3972dac49538e532 not found: ID does not exist"
Jul 28 17:30:30.910836 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.910806    2559 scope.go:117] "RemoveContainer" containerID="d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"
Jul 28 17:30:30.911051 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:30.911036    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30\": container with ID starting with d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30 not found: ID does not exist" containerID="d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"
Jul 28 17:30:30.911099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.911055    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30"} err="failed to get container status \"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30\": rpc error: code = NotFound desc = could not find container \"d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30\": container with ID starting with d2efac12f416356fb2f4b1f09eec77b21821b4e9e1c606eb9f6a2ed23b7bbc30 not found: ID does not exist"
Jul 28 17:30:30.917290 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.917266    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj"]
Jul 28 17:30:30.939928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.939901    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:30:30.940265 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940252    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940267    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940277    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940283    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940293    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="storage-initializer"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940298    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="storage-initializer"
Jul 28 17:30:30.940311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940308    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940313    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940325    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="storage-initializer"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940331    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="storage-initializer"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940338    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940343    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940351    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940355    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940362    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="storage-initializer"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940367    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="storage-initializer"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940372    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940377    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940426    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940434    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940440    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940445    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce11840e-1649-47b0-bfef-f9b1774706f1" containerName="kube-rbac-proxy"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940452    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" containerName="kserve-container"
Jul 28 17:30:30.940500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.940458    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" containerName="kube-rbac-proxy"
Jul 28 17:30:30.945151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.945136    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:30.948075 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948058    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:30:30.948243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:30:30.948317 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948250    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:30:30.948621 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948603    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:30:30.948744 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948729    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:30:30.948856 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.948844    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:30:30.949052 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.949040    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:30:30.952133 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.952109    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:30:30.955593 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.955576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:30:30.955675 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.955658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:30.957714 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.957699    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:30:30.969930 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:30.969905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:30:31.006638 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.006768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.006768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.006768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.006768 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7mk\" (UniqueName: \"kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.006990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqm6\" (UniqueName: \"kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.006990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.006990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.006990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.006990 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.006960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107578 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqm6\" (UniqueName: \"kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107578 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.107578 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.107852 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:31.107802    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:30:31.108212 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:30:31.107911    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls podName:8b13f313-de36-474a-b3cd-006c75b00706 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:31.607889967 +0000 UTC m=+1270.490297074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls") pod "isvc-canary-promote-predictor-5d7dcd97d5-c87qb" (UID: "8b13f313-de36-474a-b3cd-006c75b00706") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:30:31.108212 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.107964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7mk\" (UniqueName: \"kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.108212 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.108386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.108448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.108448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.108558 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.108799 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.108776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.110646 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.110626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.118994 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.118974    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7mk\" (UniqueName: \"kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.119093 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.119023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqm6\" (UniqueName: \"kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-hgz9z\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.264744 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.264714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:31.402084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.402058    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:30:31.403881 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:30:31.403840    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9fcac2_a80b_4ec8_8c7c_a694beac544f.slice/crio-d3b4918dd4f986b54c21d0d0ef3655500a1276c3da7d46fb2f0041744e186e2d WatchSource:0}: Error finding container d3b4918dd4f986b54c21d0d0ef3655500a1276c3da7d46fb2f0041744e186e2d: Status 404 returned error can't find the container with id d3b4918dd4f986b54c21d0d0ef3655500a1276c3da7d46fb2f0041744e186e2d
Jul 28 17:30:31.611818 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.611787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.614149 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.614131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-c87qb\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.673322 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.673243    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebc4a67-4112-4b34-be98-97993e3b6619" path="/var/lib/kubelet/pods/7ebc4a67-4112-4b34-be98-97993e3b6619/volumes"
Jul 28 17:30:31.673729 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.673716    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d83cba32-650c-4bc3-8c0f-f509fa0fdff4" path="/var/lib/kubelet/pods/d83cba32-650c-4bc3-8c0f-f509fa0fdff4/volumes"
Jul 28 17:30:31.855580 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.855553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:31.868796 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.868762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerStarted","Data":"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"}
Jul 28 17:30:31.868973 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:31.868835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerStarted","Data":"d3b4918dd4f986b54c21d0d0ef3655500a1276c3da7d46fb2f0041744e186e2d"}
Jul 28 17:30:32.007088 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:32.007068    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:30:32.009952 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:30:32.009925    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b13f313_de36_474a_b3cd_006c75b00706.slice/crio-fa2640803d1853d0dc43791a4bcee120c869cf66f41837242a7ec571b3a38a61 WatchSource:0}: Error finding container fa2640803d1853d0dc43791a4bcee120c869cf66f41837242a7ec571b3a38a61: Status 404 returned error can't find the container with id fa2640803d1853d0dc43791a4bcee120c869cf66f41837242a7ec571b3a38a61
Jul 28 17:30:32.874311 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:32.874279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerStarted","Data":"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"}
Jul 28 17:30:32.874709 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:32.874338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerStarted","Data":"fa2640803d1853d0dc43791a4bcee120c869cf66f41837242a7ec571b3a38a61"}
Jul 28 17:30:35.890289 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:35.890256    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b13f313-de36-474a-b3cd-006c75b00706" containerID="5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734" exitCode=0
Jul 28 17:30:35.890713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:35.890328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerDied","Data":"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"}
Jul 28 17:30:35.891983 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:35.891966    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerID="8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff" exitCode=0
Jul 28 17:30:35.892203 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:35.891989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerDied","Data":"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"}
Jul 28 17:30:36.899761 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.899729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerStarted","Data":"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"}
Jul 28 17:30:36.900216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.899804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerStarted","Data":"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"}
Jul 28 17:30:36.900216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.900026    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:36.901697 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.901676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerStarted","Data":"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"}
Jul 28 17:30:36.901795 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.901722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerStarted","Data":"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"}
Jul 28 17:30:36.901915 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.901898    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:36.901951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.901945    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:36.903409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.903385    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:30:36.921605 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.921552    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podStartSLOduration=3.092263254 podStartE2EDuration="6.921533138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829269884s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:36.919386132 +0000 UTC m=+1275.801793248" watchObservedRunningTime="2026-07-28 17:30:36.921533138 +0000 UTC m=+1275.803940254"
Jul 28 17:30:36.941234 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:36.941190    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podStartSLOduration=3.226349007 podStartE2EDuration="6.941158879s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714809872s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:36.940444636 +0000 UTC m=+1275.822851762" watchObservedRunningTime="2026-07-28 17:30:36.941158879 +0000 UTC m=+1275.823565996"
Jul 28 17:30:37.906493 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:37.906447    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:30:37.906927 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:37.906506    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:37.907654 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:37.907633    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:30:38.911069 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:38.911032    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:30:42.910408 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:42.910378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:30:42.910848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:42.910826    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:30:43.915767 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:43.915742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:30:43.916224 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:43.916200    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:30:52.911078 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:52.911037    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:30:53.917105 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:30:53.917070    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:02.911113 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:02.911069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:31:03.916797 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:03.916761    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:12.911296 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:12.911259    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:31:13.916271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:13.916230    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:22.911093 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:22.911052    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:31:23.916241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:23.916205    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:32.911122 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:32.911084    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:31:33.916827 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:33.916791    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:42.911766 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:42.911687    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:31:43.916419 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:43.916380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:31:52.912037 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:52.911999    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:31:53.917708 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:31:53.917681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:32:01.012951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:01.012919    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:32:01.013458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:01.013185    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" containerID="cri-o://9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4" gracePeriod=30
Jul 28 17:32:01.013458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:01.013204    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kube-rbac-proxy" containerID="cri-o://7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527" gracePeriod=30
Jul 28 17:32:01.291388 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:01.291294    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b13f313-de36-474a-b3cd-006c75b00706" containerID="7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527" exitCode=2
Jul 28 17:32:01.291546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:01.291376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerDied","Data":"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"}
Jul 28 17:32:02.907273 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:02.907227    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 28 17:32:02.911623 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:02.911597    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 17:32:05.655761 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.655733    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:32:05.788317 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx7mk\" (UniqueName: \"kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk\") pod \"8b13f313-de36-474a-b3cd-006c75b00706\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") "
Jul 28 17:32:05.788478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788316    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") pod \"8b13f313-de36-474a-b3cd-006c75b00706\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") "
Jul 28 17:32:05.788478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788353    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert\") pod \"8b13f313-de36-474a-b3cd-006c75b00706\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") "
Jul 28 17:32:05.788478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config\") pod \"8b13f313-de36-474a-b3cd-006c75b00706\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") "
Jul 28 17:32:05.788478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location\") pod \"8b13f313-de36-474a-b3cd-006c75b00706\" (UID: \"8b13f313-de36-474a-b3cd-006c75b00706\") "
Jul 28 17:32:05.788795 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b13f313-de36-474a-b3cd-006c75b00706" (UID: "8b13f313-de36-474a-b3cd-006c75b00706"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:05.788890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788677    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b13f313-de36-474a-b3cd-006c75b00706" (UID: "8b13f313-de36-474a-b3cd-006c75b00706"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:05.788890 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.788799    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b13f313-de36-474a-b3cd-006c75b00706" (UID: "8b13f313-de36-474a-b3cd-006c75b00706"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:05.790353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.790326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk" (OuterVolumeSpecName: "kube-api-access-tx7mk") pod "8b13f313-de36-474a-b3cd-006c75b00706" (UID: "8b13f313-de36-474a-b3cd-006c75b00706"). InnerVolumeSpecName "kube-api-access-tx7mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:05.790462 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.790395    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b13f313-de36-474a-b3cd-006c75b00706" (UID: "8b13f313-de36-474a-b3cd-006c75b00706"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:05.889284 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.889254    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b13f313-de36-474a-b3cd-006c75b00706-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:32:05.889284 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.889280    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7mk\" (UniqueName: \"kubernetes.io/projected/8b13f313-de36-474a-b3cd-006c75b00706-kube-api-access-tx7mk\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:32:05.889454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.889293    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b13f313-de36-474a-b3cd-006c75b00706-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:32:05.889454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.889307    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:32:05.889454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:05.889319    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b13f313-de36-474a-b3cd-006c75b00706-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:32:06.318196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.318160    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b13f313-de36-474a-b3cd-006c75b00706" containerID="9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4" exitCode=0
Jul 28 17:32:06.318382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.318216    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"
Jul 28 17:32:06.318382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.318240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerDied","Data":"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"}
Jul 28 17:32:06.318382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.318296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb" event={"ID":"8b13f313-de36-474a-b3cd-006c75b00706","Type":"ContainerDied","Data":"fa2640803d1853d0dc43791a4bcee120c869cf66f41837242a7ec571b3a38a61"}
Jul 28 17:32:06.318382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.318325    2559 scope.go:117] "RemoveContainer" containerID="7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"
Jul 28 17:32:06.327827 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.327811    2559 scope.go:117] "RemoveContainer" containerID="9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"
Jul 28 17:32:06.335937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.335919    2559 scope.go:117] "RemoveContainer" containerID="5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"
Jul 28 17:32:06.344353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.343250    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:32:06.346718 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.346699    2559 scope.go:117] "RemoveContainer" containerID="7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"
Jul 28 17:32:06.347030 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:32:06.347011    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527\": container with ID starting with 7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527 not found: ID does not exist" containerID="7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"
Jul 28 17:32:06.347079 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347039    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527"} err="failed to get container status \"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527\": rpc error: code = NotFound desc = could not find container \"7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527\": container with ID starting with 7729fab7944c3c98d72cbdd50098f991e54881b983f65bc9f202fa1138b88527 not found: ID does not exist"
Jul 28 17:32:06.347079 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347059    2559 scope.go:117] "RemoveContainer" containerID="9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"
Jul 28 17:32:06.347324 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:32:06.347302    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4\": container with ID starting with 9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4 not found: ID does not exist" containerID="9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"
Jul 28 17:32:06.347375 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347332    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4"} err="failed to get container status \"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4\": rpc error: code = NotFound desc = could not find container \"9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4\": container with ID starting with 9114710e4ce0158ed5a2f81d23a91d2b896c28e4a99ce85315246ce9147dc5b4 not found: ID does not exist"
Jul 28 17:32:06.347375 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347354    2559 scope.go:117] "RemoveContainer" containerID="5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"
Jul 28 17:32:06.347583 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:32:06.347566    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734\": container with ID starting with 5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734 not found: ID does not exist" containerID="5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"
Jul 28 17:32:06.347650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347600    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734"} err="failed to get container status \"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734\": rpc error: code = NotFound desc = could not find container \"5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734\": container with ID starting with 5e1c63c451a76497f6e3a8cd822e38125b8f7a8297c2e0a42a0c7cc9f1fd2734 not found: ID does not exist"
Jul 28 17:32:06.347693 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:06.347655    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb"]
Jul 28 17:32:07.672769 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:32:07.672740    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b13f313-de36-474a-b3cd-006c75b00706" path="/var/lib/kubelet/pods/8b13f313-de36-474a-b3cd-006c75b00706/volumes"
Jul 28 17:33:41.296574 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:41.296533    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:33:41.297143 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:41.296901    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" containerID="cri-o://b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679" gracePeriod=30
Jul 28 17:33:41.297143 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:41.296910    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kube-rbac-proxy" containerID="cri-o://a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1" gracePeriod=30
Jul 28 17:33:41.757240 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:41.757209    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerID="a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1" exitCode=2
Jul 28 17:33:41.757410 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:41.757269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerDied","Data":"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"}
Jul 28 17:33:43.912077 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:43.912039    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 28 17:33:43.916803 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:43.916779    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 28 17:33:45.942088 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:45.942061    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:33:46.001996 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.001912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert\") pod \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") "
Jul 28 17:33:46.001996 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.001966    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls\") pod \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") "
Jul 28 17:33:46.002241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkqm6\" (UniqueName: \"kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6\") pod \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") "
Jul 28 17:33:46.002241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002042    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location\") pod \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") "
Jul 28 17:33:46.002241 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config\") pod \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\" (UID: \"7b9fcac2-a80b-4ec8-8c7c-a694beac544f\") "
Jul 28 17:33:46.002414 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002283    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b9fcac2-a80b-4ec8-8c7c-a694beac544f" (UID: "7b9fcac2-a80b-4ec8-8c7c-a694beac544f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:46.002461 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002445    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b9fcac2-a80b-4ec8-8c7c-a694beac544f" (UID: "7b9fcac2-a80b-4ec8-8c7c-a694beac544f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:46.002534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.002507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b9fcac2-a80b-4ec8-8c7c-a694beac544f" (UID: "7b9fcac2-a80b-4ec8-8c7c-a694beac544f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:46.004210 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.004182    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6" (OuterVolumeSpecName: "kube-api-access-pkqm6") pod "7b9fcac2-a80b-4ec8-8c7c-a694beac544f" (UID: "7b9fcac2-a80b-4ec8-8c7c-a694beac544f"). InnerVolumeSpecName "kube-api-access-pkqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:46.004287 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.004186    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b9fcac2-a80b-4ec8-8c7c-a694beac544f" (UID: "7b9fcac2-a80b-4ec8-8c7c-a694beac544f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:46.103466 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.103442    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:33:46.103466 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.103462    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:33:46.103466 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.103471    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:33:46.103660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.103479    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqm6\" (UniqueName: \"kubernetes.io/projected/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kube-api-access-pkqm6\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:33:46.103660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.103488    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9fcac2-a80b-4ec8-8c7c-a694beac544f-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:33:46.495889 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.495843    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:33:46.496239 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496226    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496241    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496252    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kube-rbac-proxy"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496258    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kube-rbac-proxy"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496266    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496274    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496288    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kube-rbac-proxy"
Jul 28 17:33:46.496297 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496293    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kube-rbac-proxy"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496303    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="storage-initializer"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496308    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="storage-initializer"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496320    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="storage-initializer"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496325    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="storage-initializer"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496387    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kserve-container"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496396    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kserve-container"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496403    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerName="kube-rbac-proxy"
Jul 28 17:33:46.496516 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.496409    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b13f313-de36-474a-b3cd-006c75b00706" containerName="kube-rbac-proxy"
Jul 28 17:33:46.499849 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.499829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.504060 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.504039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:33:46.504568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.504542    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:33:46.505893 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.505863    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:33:46.509238 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.509223    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.515975 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.511746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:33:46.515975 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.514477    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:33:46.524455 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.524433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:33:46.606650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.606841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926dp\" (UniqueName: \"kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.607196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xtd\" (UniqueName: \"kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.607196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.607196 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.606975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.707960 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.707934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.707975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.707996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708119 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708377 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926dp\" (UniqueName: \"kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708377 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xtd\" (UniqueName: \"kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708377 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708377 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:33:46.708271    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:33:46.708377 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:33:46.708356    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls podName:d112fa48-4a4f-4a2f-bc4d-8065cb437529 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:47.208334208 +0000 UTC m=+1466.090741303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls") pod "isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:33:46.708645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708645 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.708817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.708817 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.708804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.710995 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.710975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.717338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.717318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926dp\" (UniqueName: \"kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp\") pod \"isvc-canary-rollback-predictor-b4645bb5c-4b62h\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.717418 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.717354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xtd\" (UniqueName: \"kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:46.783192 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.783133    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" containerID="b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679" exitCode=0
Jul 28 17:33:46.783192 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.783176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerDied","Data":"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"}
Jul 28 17:33:46.783347 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.783204    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"
Jul 28 17:33:46.783347 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.783223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z" event={"ID":"7b9fcac2-a80b-4ec8-8c7c-a694beac544f","Type":"ContainerDied","Data":"d3b4918dd4f986b54c21d0d0ef3655500a1276c3da7d46fb2f0041744e186e2d"}
Jul 28 17:33:46.783347 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.783249    2559 scope.go:117] "RemoveContainer" containerID="a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"
Jul 28 17:33:46.792325 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.792302    2559 scope.go:117] "RemoveContainer" containerID="b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"
Jul 28 17:33:46.799676 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.799661    2559 scope.go:117] "RemoveContainer" containerID="8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"
Jul 28 17:33:46.806134 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.806112    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:33:46.806794 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.806780    2559 scope.go:117] "RemoveContainer" containerID="a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"
Jul 28 17:33:46.807063 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:33:46.807046    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1\": container with ID starting with a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1 not found: ID does not exist" containerID="a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"
Jul 28 17:33:46.807118 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.807071    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1"} err="failed to get container status \"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1\": rpc error: code = NotFound desc = could not find container \"a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1\": container with ID starting with a899c6313bcdbed4aaef5fdb040e2708f2a8fdbc4e7de76c66dec01ffb454cd1 not found: ID does not exist"
Jul 28 17:33:46.807118 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.807088    2559 scope.go:117] "RemoveContainer" containerID="b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"
Jul 28 17:33:46.807309 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:33:46.807295    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679\": container with ID starting with b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679 not found: ID does not exist" containerID="b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"
Jul 28 17:33:46.807353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.807313    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679"} err="failed to get container status \"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679\": rpc error: code = NotFound desc = could not find container \"b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679\": container with ID starting with b29a2b0f8c79494e6a8d59c0ec654282a9362d5faea8d3567bbb7ce9487e1679 not found: ID does not exist"
Jul 28 17:33:46.807353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.807325    2559 scope.go:117] "RemoveContainer" containerID="8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"
Jul 28 17:33:46.807519 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:33:46.807504    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff\": container with ID starting with 8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff not found: ID does not exist" containerID="8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"
Jul 28 17:33:46.807559 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.807523    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff"} err="failed to get container status \"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff\": rpc error: code = NotFound desc = could not find container \"8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff\": container with ID starting with 8b46219d67e7e76d0d16b0717cda342a152be5c06fe8b462cf7791d0dd28f6ff not found: ID does not exist"
Jul 28 17:33:46.810357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.810338    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z"]
Jul 28 17:33:46.810455 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.810409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:46.941507 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:46.941461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:33:47.213272 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.213245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:47.215556 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.215530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:47.427366 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.427336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:47.559444 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.559417    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:33:47.560796 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:33:47.560770    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd112fa48_4a4f_4a2f_bc4d_8065cb437529.slice/crio-4f150c7f2db5361e3abf7d20861d7d24395eeaf39fe0713cab426901422e3e5e WatchSource:0}: Error finding container 4f150c7f2db5361e3abf7d20861d7d24395eeaf39fe0713cab426901422e3e5e: Status 404 returned error can't find the container with id 4f150c7f2db5361e3abf7d20861d7d24395eeaf39fe0713cab426901422e3e5e
Jul 28 17:33:47.674950 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.674917    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9fcac2-a80b-4ec8-8c7c-a694beac544f" path="/var/lib/kubelet/pods/7b9fcac2-a80b-4ec8-8c7c-a694beac544f/volumes"
Jul 28 17:33:47.789699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.789601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerStarted","Data":"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"}
Jul 28 17:33:47.789699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.789680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerStarted","Data":"0fb1b3cf8e4c92f312ec24e4c5e620141bd254ccfc652b2dc0e8ef3f3eeb0e05"}
Jul 28 17:33:47.792060 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.792037    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerStarted","Data":"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"}
Jul 28 17:33:47.792186 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:47.792080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerStarted","Data":"4f150c7f2db5361e3abf7d20861d7d24395eeaf39fe0713cab426901422e3e5e"}
Jul 28 17:33:50.806202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:50.806126    2559 generic.go:356] "Generic (PLEG): container finished" podID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerID="3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929" exitCode=0
Jul 28 17:33:50.806202 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:50.806175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerDied","Data":"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"}
Jul 28 17:33:51.814856 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.814822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerStarted","Data":"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"}
Jul 28 17:33:51.815336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.814920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerStarted","Data":"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"}
Jul 28 17:33:51.815336 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.815132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:51.816259 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.816233    2559 generic.go:356] "Generic (PLEG): container finished" podID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerID="a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57" exitCode=0
Jul 28 17:33:51.816349 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.816267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerDied","Data":"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"}
Jul 28 17:33:51.834842 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:51.834803    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podStartSLOduration=2.315664392 podStartE2EDuration="5.83478884s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.519124448s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:51.833313128 +0000 UTC m=+1470.715720262" watchObservedRunningTime="2026-07-28 17:33:51.83478884 +0000 UTC m=+1470.717195956"
Jul 28 17:33:52.823684 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.823643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerStarted","Data":"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"}
Jul 28 17:33:52.824151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.823724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerStarted","Data":"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"}
Jul 28 17:33:52.824151 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.823894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:52.824286 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.824268    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:52.824332 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.824317    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:52.825186 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.825161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:33:52.825501 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.825472    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:33:52.843886 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:52.843830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podStartSLOduration=3.023672582 podStartE2EDuration="6.843816077s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820143495s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:52.84235954 +0000 UTC m=+1471.724766656" watchObservedRunningTime="2026-07-28 17:33:52.843816077 +0000 UTC m=+1471.726223192"
Jul 28 17:33:53.828665 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:53.828624    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:33:53.829116 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:53.828774    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:33:58.832752 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:58.832720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:33:58.833265 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:58.833238    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:33:58.833888 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:58.833846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:33:58.834525 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:33:58.834496    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:08.833143 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:08.833107    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:08.835351 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:08.835329    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:18.833454 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:18.833413    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:18.834622 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:18.834599    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:21.625256 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:21.625227    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:34:21.628146 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:21.628123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:34:28.833792 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:28.833749    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:28.834987 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:28.834966    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:38.833211 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:38.833176    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:38.834378 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:38.834356    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:48.833370 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:48.833318    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:48.835736 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:48.834702    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:34:58.833820 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:58.833782    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:34:58.835010 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:34:58.834988    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:35:03.668057 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:03.668017    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 17:35:08.835044 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:08.835018    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:35:13.672432 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:13.672406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:35:16.534969 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:16.534931    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:35:16.535427 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:16.535322    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container" containerID="cri-o://14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe" gracePeriod=30
Jul 28 17:35:16.535776 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:16.535745    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kube-rbac-proxy" containerID="cri-o://ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9" gracePeriod=30
Jul 28 17:35:17.226943 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:17.226910    2559 generic.go:356] "Generic (PLEG): container finished" podID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerID="ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9" exitCode=2
Jul 28 17:35:17.227118 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:17.226983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerDied","Data":"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"}
Jul 28 17:35:18.828840 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:18.828789    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 28 17:35:20.983360 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:20.983333    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:35:21.007619 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location\") pod \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") "
Jul 28 17:35:21.007808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007641    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert\") pod \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") "
Jul 28 17:35:21.007808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007677    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config\") pod \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") "
Jul 28 17:35:21.007808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007728    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") pod \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") "
Jul 28 17:35:21.007808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007777    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2xtd\" (UniqueName: \"kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd\") pod \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\" (UID: \"d112fa48-4a4f-4a2f-bc4d-8065cb437529\") "
Jul 28 17:35:21.008084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.007972    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d112fa48-4a4f-4a2f-bc4d-8065cb437529" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:21.008084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.008034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d112fa48-4a4f-4a2f-bc4d-8065cb437529" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:21.008084 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.008065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d112fa48-4a4f-4a2f-bc4d-8065cb437529" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:21.008218 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.008122    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:35:21.008218 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.008142    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:35:21.010047 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.010013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd" (OuterVolumeSpecName: "kube-api-access-m2xtd") pod "d112fa48-4a4f-4a2f-bc4d-8065cb437529" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529"). InnerVolumeSpecName "kube-api-access-m2xtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:21.010143 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.010077    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d112fa48-4a4f-4a2f-bc4d-8065cb437529" (UID: "d112fa48-4a4f-4a2f-bc4d-8065cb437529"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:21.109350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.109290    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d112fa48-4a4f-4a2f-bc4d-8065cb437529-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:35:21.109350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.109315    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2xtd\" (UniqueName: \"kubernetes.io/projected/d112fa48-4a4f-4a2f-bc4d-8065cb437529-kube-api-access-m2xtd\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:35:21.109350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.109325    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d112fa48-4a4f-4a2f-bc4d-8065cb437529-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:35:21.246340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.246307    2559 generic.go:356] "Generic (PLEG): container finished" podID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerID="14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe" exitCode=0
Jul 28 17:35:21.246503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.246360    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"
Jul 28 17:35:21.246503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.246413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerDied","Data":"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"}
Jul 28 17:35:21.246503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.246463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d" event={"ID":"d112fa48-4a4f-4a2f-bc4d-8065cb437529","Type":"ContainerDied","Data":"4f150c7f2db5361e3abf7d20861d7d24395eeaf39fe0713cab426901422e3e5e"}
Jul 28 17:35:21.246503 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.246495    2559 scope.go:117] "RemoveContainer" containerID="ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"
Jul 28 17:35:21.255696 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.255680    2559 scope.go:117] "RemoveContainer" containerID="14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"
Jul 28 17:35:21.263698 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.263683    2559 scope.go:117] "RemoveContainer" containerID="a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"
Jul 28 17:35:21.270992 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.270968    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:35:21.271931 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.271886    2559 scope.go:117] "RemoveContainer" containerID="ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"
Jul 28 17:35:21.272188 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:35:21.272167    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9\": container with ID starting with ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9 not found: ID does not exist" containerID="ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"
Jul 28 17:35:21.272242 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.272200    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9"} err="failed to get container status \"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9\": rpc error: code = NotFound desc = could not find container \"ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9\": container with ID starting with ad791a6a803e00f3b5aa13bb653cee5274c90c333fdfc058a5683099ef4b64c9 not found: ID does not exist"
Jul 28 17:35:21.272242 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.272222    2559 scope.go:117] "RemoveContainer" containerID="14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"
Jul 28 17:35:21.272448 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:35:21.272434    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe\": container with ID starting with 14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe not found: ID does not exist" containerID="14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"
Jul 28 17:35:21.272497 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.272453    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe"} err="failed to get container status \"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe\": rpc error: code = NotFound desc = could not find container \"14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe\": container with ID starting with 14f9a19b043f87cfedd2acd073be53f9bdc02aee5febcf02f2131eeb73e4fabe not found: ID does not exist"
Jul 28 17:35:21.272497 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.272465    2559 scope.go:117] "RemoveContainer" containerID="a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"
Jul 28 17:35:21.272684 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:35:21.272664    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57\": container with ID starting with a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57 not found: ID does not exist" containerID="a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"
Jul 28 17:35:21.272730 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.272691    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57"} err="failed to get container status \"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57\": rpc error: code = NotFound desc = could not find container \"a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57\": container with ID starting with a68428d2a9f82a5055ee3545eb3f2920e7ba8e80e146bca6388a9c5daf960f57 not found: ID does not exist"
Jul 28 17:35:21.275559 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.275534    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d"]
Jul 28 17:35:21.673006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:35:21.672976    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" path="/var/lib/kubelet/pods/d112fa48-4a4f-4a2f-bc4d-8065cb437529/volumes"
Jul 28 17:36:36.653236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:36.653200    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:36:36.653807 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:36.653506    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" containerID="cri-o://9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559" gracePeriod=30
Jul 28 17:36:36.653807 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:36.653521    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kube-rbac-proxy" containerID="cri-o://36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116" gracePeriod=30
Jul 28 17:36:36.676257 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:36.676229    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa39f9c_bb5e_46dc_bbc4_33bca084a22d.slice/crio-36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 17:36:37.591544 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:37.591514    2559 generic.go:356] "Generic (PLEG): container finished" podID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerID="36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116" exitCode=2
Jul 28 17:36:37.591735 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:37.591590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerDied","Data":"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"}
Jul 28 17:36:38.829107 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:38.829062    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 28 17:36:38.834534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:38.834510    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 17:36:41.203798 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.203777    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:36:41.265884 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.265848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location\") pod \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") "
Jul 28 17:36:41.266031 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.265903    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config\") pod \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") "
Jul 28 17:36:41.266031 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.265931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-926dp\" (UniqueName: \"kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp\") pod \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") "
Jul 28 17:36:41.266147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert\") pod \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") "
Jul 28 17:36:41.266147 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266086    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls\") pod \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\" (UID: \"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d\") "
Jul 28 17:36:41.266252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" (UID: "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:41.266309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266270    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" (UID: "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:41.266405 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266389    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:36:41.266458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266407    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:36:41.266458 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.266405    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" (UID: "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:41.268131 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.268109    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp" (OuterVolumeSpecName: "kube-api-access-926dp") pod "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" (UID: "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d"). InnerVolumeSpecName "kube-api-access-926dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:41.268194 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.268145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" (UID: "6aa39f9c-bb5e-46dc-bbc4-33bca084a22d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:41.367223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.367195    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:36:41.367223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.367219    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:36:41.367223 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.367228    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926dp\" (UniqueName: \"kubernetes.io/projected/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d-kube-api-access-926dp\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:36:41.612145 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.612115    2559 generic.go:356] "Generic (PLEG): container finished" podID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerID="9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559" exitCode=0
Jul 28 17:36:41.612306 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.612171    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"
Jul 28 17:36:41.612306 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.612179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerDied","Data":"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"}
Jul 28 17:36:41.612306 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.612240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h" event={"ID":"6aa39f9c-bb5e-46dc-bbc4-33bca084a22d","Type":"ContainerDied","Data":"0fb1b3cf8e4c92f312ec24e4c5e620141bd254ccfc652b2dc0e8ef3f3eeb0e05"}
Jul 28 17:36:41.612306 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.612278    2559 scope.go:117] "RemoveContainer" containerID="36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"
Jul 28 17:36:41.626478 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.626458    2559 scope.go:117] "RemoveContainer" containerID="9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"
Jul 28 17:36:41.634231 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.634210    2559 scope.go:117] "RemoveContainer" containerID="3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"
Jul 28 17:36:41.639042 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.639023    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:36:41.641974 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.641950    2559 scope.go:117] "RemoveContainer" containerID="36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"
Jul 28 17:36:41.642216 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:41.642199    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116\": container with ID starting with 36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116 not found: ID does not exist" containerID="36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"
Jul 28 17:36:41.642271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.642224    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116"} err="failed to get container status \"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116\": rpc error: code = NotFound desc = could not find container \"36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116\": container with ID starting with 36f070ce3837b2d87833267a9e0be63444e974e84d25fef8f1e10007b90a9116 not found: ID does not exist"
Jul 28 17:36:41.642271 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.642241    2559 scope.go:117] "RemoveContainer" containerID="9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"
Jul 28 17:36:41.642450 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:41.642435    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559\": container with ID starting with 9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559 not found: ID does not exist" containerID="9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"
Jul 28 17:36:41.642513 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.642453    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559"} err="failed to get container status \"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559\": rpc error: code = NotFound desc = could not find container \"9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559\": container with ID starting with 9a31fe2184efd316396634d2e03697530c9d2eb9fa1702a248a53956fb87e559 not found: ID does not exist"
Jul 28 17:36:41.642513 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.642466    2559 scope.go:117] "RemoveContainer" containerID="3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"
Jul 28 17:36:41.642804 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:41.642772    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929\": container with ID starting with 3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929 not found: ID does not exist" containerID="3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"
Jul 28 17:36:41.642950 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.642804    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929"} err="failed to get container status \"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929\": rpc error: code = NotFound desc = could not find container \"3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929\": container with ID starting with 3ea18dec638d5a731ae5ff8db790eedad667cf86c4f726f2769bbff96b238929 not found: ID does not exist"
Jul 28 17:36:41.644600 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.644582    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h"]
Jul 28 17:36:41.672326 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.672297    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" path="/var/lib/kubelet/pods/6aa39f9c-bb5e-46dc-bbc4-33bca084a22d/volumes"
Jul 28 17:36:41.862616 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862589    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:36:41.862944 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862931    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kube-rbac-proxy"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862945    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kube-rbac-proxy"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862958    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862970    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kube-rbac-proxy"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862975    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kube-rbac-proxy"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862982    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.862987    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container"
Jul 28 17:36:41.863002 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863003    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="storage-initializer"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863008    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="storage-initializer"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863016    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="storage-initializer"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863021    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="storage-initializer"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863092    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kube-rbac-proxy"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863101    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa39f9c-bb5e-46dc-bbc4-33bca084a22d" containerName="kserve-container"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863109    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kserve-container"
Jul 28 17:36:41.863243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.863114    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d112fa48-4a4f-4a2f-bc4d-8065cb437529" containerName="kube-rbac-proxy"
Jul 28 17:36:41.867560 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.867544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:41.869857 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.869832    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:36:41.869857 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.869840    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:36:41.870137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.870117    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:36:41.870200 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.870157    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:36:41.870200 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.870188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:36:41.870414 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.870397    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:36:41.870514 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.870434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:36:41.876790 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.876772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:36:41.906224 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.906201    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:36:41.909829 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.909814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.911996 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.911981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 17:36:41.921567 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.921548    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:36:41.971481 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.971660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.971660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:41.971660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.971660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:41.971660 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.971833 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:41.971833 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbng\" (UniqueName: \"kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:41.971833 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86l2b\" (UniqueName: \"kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:41.971833 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:41.971795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.072825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.072791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.072825 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.072830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.072850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.072911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.072941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbng\" (UniqueName: \"kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86l2b\" (UniqueName: \"kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073569 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073569 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:42.073543    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:36:42.073677 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:36:42.073606    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls podName:6d6a1647-5e02-4040-8217-cf9443531a73 nodeName:}" failed. No retries permitted until 2026-07-28 17:36:42.573586979 +0000 UTC m=+1641.455994075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls") pod "isvc-canary-stop-predictor-5bf5667989-dv7nc" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:36:42.073747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.073747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.073941 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.074022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.073968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.075556 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.075528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.083361 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.083340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbng\" (UniqueName: \"kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.083903 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.083859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86l2b\" (UniqueName: \"kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.219801 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.219715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:42.358229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.358207    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:36:42.360652 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:36:42.360624    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1a3fac_29bd_4a82_873a_fdaec9e7ec0a.slice/crio-9038a21de4e041e522cf7e90a620239a11a67f13e63211c008dd9ca3937ca161 WatchSource:0}: Error finding container 9038a21de4e041e522cf7e90a620239a11a67f13e63211c008dd9ca3937ca161: Status 404 returned error can't find the container with id 9038a21de4e041e522cf7e90a620239a11a67f13e63211c008dd9ca3937ca161
Jul 28 17:36:42.576701 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.576673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.579187 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.579164    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-dv7nc\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.619582 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.619548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerStarted","Data":"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"}
Jul 28 17:36:42.619756 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.619627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerStarted","Data":"9038a21de4e041e522cf7e90a620239a11a67f13e63211c008dd9ca3937ca161"}
Jul 28 17:36:42.779245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.779200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:42.918152 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:42.918117    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:36:42.921939 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:36:42.921912    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6a1647_5e02_4040_8217_cf9443531a73.slice/crio-566fb3298aea0759dcd83a9a1d928de1440ff8c08204b050a6332b54e3fd5a38 WatchSource:0}: Error finding container 566fb3298aea0759dcd83a9a1d928de1440ff8c08204b050a6332b54e3fd5a38: Status 404 returned error can't find the container with id 566fb3298aea0759dcd83a9a1d928de1440ff8c08204b050a6332b54e3fd5a38
Jul 28 17:36:43.627095 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:43.627057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerStarted","Data":"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"}
Jul 28 17:36:43.627529 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:43.627131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerStarted","Data":"566fb3298aea0759dcd83a9a1d928de1440ff8c08204b050a6332b54e3fd5a38"}
Jul 28 17:36:46.642248 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:46.642215    2559 generic.go:356] "Generic (PLEG): container finished" podID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerID="6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc" exitCode=0
Jul 28 17:36:46.642727 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:46.642294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerDied","Data":"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"}
Jul 28 17:36:47.648028 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.647998    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d6a1647-5e02-4040-8217-cf9443531a73" containerID="15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450" exitCode=0
Jul 28 17:36:47.648470 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.648064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerDied","Data":"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"}
Jul 28 17:36:47.650231 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.650212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerStarted","Data":"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"}
Jul 28 17:36:47.650328 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.650259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerStarted","Data":"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"}
Jul 28 17:36:47.650500 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.650483    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:47.650587 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.650526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:47.652032 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.652008    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:36:47.686448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:47.686404    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podStartSLOduration=2.8165882570000003 podStartE2EDuration="6.686389662s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869801405s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:47.685017339 +0000 UTC m=+1646.567424481" watchObservedRunningTime="2026-07-28 17:36:47.686389662 +0000 UTC m=+1646.568796778"
Jul 28 17:36:48.656557 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:48.656528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerStarted","Data":"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"}
Jul 28 17:36:48.657024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:48.656583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerStarted","Data":"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"}
Jul 28 17:36:48.657024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:48.656822    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:36:48.657024 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:48.656895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:48.676136 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:48.676089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podStartSLOduration=3.8070934149999998 podStartE2EDuration="7.676073459s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868980044s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:48.674380258 +0000 UTC m=+1647.556787373" watchObservedRunningTime="2026-07-28 17:36:48.676073459 +0000 UTC m=+1647.558480575"
Jul 28 17:36:49.662216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:49.662190    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:49.663426 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:49.663399    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:36:50.667446 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:50.667403    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:36:53.661576 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:53.661545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:36:53.662260 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:53.662230    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:36:55.673501 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:55.673475    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:36:55.673995 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:36:55.673971    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:03.662712 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:03.662674    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:05.674103 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:05.674064    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:13.662281 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:13.662244    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:15.674262 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:15.674227    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:23.662063 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:23.662024    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:25.674851 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:25.674811    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:33.663106 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:33.663064    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:35.674722 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:35.674685    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:43.662691 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:43.662603    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:45.674441 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:45.674405    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:37:53.662711 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:53.662665    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:54.667067 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:54.667029    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:37:55.674950 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:37:55.674911    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:38:04.668043 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:04.668012    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:38:05.674631 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:05.674599    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:38:11.913943 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.913910    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:38:11.914612 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.914145    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" containerID="cri-o://daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27" gracePeriod=30
Jul 28 17:38:11.914612 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.914185    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kube-rbac-proxy" containerID="cri-o://fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96" gracePeriod=30
Jul 28 17:38:11.945305 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.945280    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:38:11.945535 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.945507    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" containerID="cri-o://bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078" gracePeriod=30
Jul 28 17:38:11.945639 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:11.945547    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kube-rbac-proxy" containerID="cri-o://2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1" gracePeriod=30
Jul 28 17:38:12.053350 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:12.053319    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d6a1647-5e02-4040-8217-cf9443531a73" containerID="fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96" exitCode=2
Jul 28 17:38:12.053456 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:12.053360    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerDied","Data":"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"}
Jul 28 17:38:13.059267 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:13.059237    2559 generic.go:356] "Generic (PLEG): container finished" podID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerID="2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1" exitCode=2
Jul 28 17:38:13.059641 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:13.059315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerDied","Data":"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"}
Jul 28 17:38:13.657976 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:13.657933    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 28 17:38:14.667103 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:14.667064    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 17:38:15.667546 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:15.667494    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 28 17:38:15.674100 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:15.674073    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 17:38:16.778475 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.778453    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:38:16.866061 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86l2b\" (UniqueName: \"kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b\") pod \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") "
Jul 28 17:38:16.866250 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls\") pod \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") "
Jul 28 17:38:16.866250 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866139    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert\") pod \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") "
Jul 28 17:38:16.866385 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config\") pod \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") "
Jul 28 17:38:16.866385 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location\") pod \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\" (UID: \"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a\") "
Jul 28 17:38:16.866620 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866593    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" (UID: "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:16.866777 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866740    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" (UID: "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:16.866777 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.866743    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" (UID: "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:16.868346 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.868321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b" (OuterVolumeSpecName: "kube-api-access-86l2b") pod "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" (UID: "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a"). InnerVolumeSpecName "kube-api-access-86l2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:16.868487 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.868471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" (UID: "2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:16.885827 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.885808    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:38:16.967167 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967135    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config\") pod \"6d6a1647-5e02-4040-8217-cf9443531a73\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") "
Jul 28 17:38:16.967329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location\") pod \"6d6a1647-5e02-4040-8217-cf9443531a73\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") "
Jul 28 17:38:16.967329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967238    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert\") pod \"6d6a1647-5e02-4040-8217-cf9443531a73\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") "
Jul 28 17:38:16.967329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967274    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") pod \"6d6a1647-5e02-4040-8217-cf9443531a73\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") "
Jul 28 17:38:16.967329 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967309    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pbng\" (UniqueName: \"kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng\") pod \"6d6a1647-5e02-4040-8217-cf9443531a73\" (UID: \"6d6a1647-5e02-4040-8217-cf9443531a73\") "
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967519    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967538    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967551    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967565    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967580    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86l2b\" (UniqueName: \"kubernetes.io/projected/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a-kube-api-access-86l2b\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d6a1647-5e02-4040-8217-cf9443531a73" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:16.967640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967630    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d6a1647-5e02-4040-8217-cf9443531a73" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:16.967935 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.967659    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d6a1647-5e02-4040-8217-cf9443531a73" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:16.969440 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.969416    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng" (OuterVolumeSpecName: "kube-api-access-4pbng") pod "6d6a1647-5e02-4040-8217-cf9443531a73" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73"). InnerVolumeSpecName "kube-api-access-4pbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:16.969539 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:16.969452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d6a1647-5e02-4040-8217-cf9443531a73" (UID: "6d6a1647-5e02-4040-8217-cf9443531a73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:17.068185 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.068161    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:17.068185 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.068183    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6a1647-5e02-4040-8217-cf9443531a73-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:17.068185 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.068192    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbng\" (UniqueName: \"kubernetes.io/projected/6d6a1647-5e02-4040-8217-cf9443531a73-kube-api-access-4pbng\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:17.068400 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.068202    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6a1647-5e02-4040-8217-cf9443531a73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:17.068400 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.068210    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6a1647-5e02-4040-8217-cf9443531a73-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:38:17.079448 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.079422    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d6a1647-5e02-4040-8217-cf9443531a73" containerID="daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27" exitCode=0
Jul 28 17:38:17.079565 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.079476    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"
Jul 28 17:38:17.079565 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.079502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerDied","Data":"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"}
Jul 28 17:38:17.079666 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.079574    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc" event={"ID":"6d6a1647-5e02-4040-8217-cf9443531a73","Type":"ContainerDied","Data":"566fb3298aea0759dcd83a9a1d928de1440ff8c08204b050a6332b54e3fd5a38"}
Jul 28 17:38:17.079666 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.079614    2559 scope.go:117] "RemoveContainer" containerID="fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"
Jul 28 17:38:17.092737 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.092714    2559 generic.go:356] "Generic (PLEG): container finished" podID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerID="bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078" exitCode=0
Jul 28 17:38:17.092879 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.092761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerDied","Data":"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"}
Jul 28 17:38:17.092879 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.092776    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"
Jul 28 17:38:17.092879 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.092841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf" event={"ID":"2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a","Type":"ContainerDied","Data":"9038a21de4e041e522cf7e90a620239a11a67f13e63211c008dd9ca3937ca161"}
Jul 28 17:38:17.098644 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.098633    2559 scope.go:117] "RemoveContainer" containerID="daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"
Jul 28 17:38:17.106465 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.106446    2559 scope.go:117] "RemoveContainer" containerID="15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"
Jul 28 17:38:17.107808 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.107789    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:38:17.112059 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.112037    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc"]
Jul 28 17:38:17.113936 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.113915    2559 scope.go:117] "RemoveContainer" containerID="fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"
Jul 28 17:38:17.114186 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.114168    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96\": container with ID starting with fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96 not found: ID does not exist" containerID="fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"
Jul 28 17:38:17.114249 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114194    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96"} err="failed to get container status \"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96\": rpc error: code = NotFound desc = could not find container \"fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96\": container with ID starting with fc6cdecc188759e931c45de814d507243362829b2f3231059d014ebc16854d96 not found: ID does not exist"
Jul 28 17:38:17.114249 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114210    2559 scope.go:117] "RemoveContainer" containerID="daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"
Jul 28 17:38:17.114440 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.114423    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27\": container with ID starting with daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27 not found: ID does not exist" containerID="daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"
Jul 28 17:38:17.114480 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114448    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27"} err="failed to get container status \"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27\": rpc error: code = NotFound desc = could not find container \"daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27\": container with ID starting with daa4c796ee8ebb4e7488e2e0bdb9218ed8dd4e34756bb4e8333b5a80bad10d27 not found: ID does not exist"
Jul 28 17:38:17.114480 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114466    2559 scope.go:117] "RemoveContainer" containerID="15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"
Jul 28 17:38:17.114705 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.114689    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450\": container with ID starting with 15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450 not found: ID does not exist" containerID="15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"
Jul 28 17:38:17.114747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114709    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450"} err="failed to get container status \"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450\": rpc error: code = NotFound desc = could not find container \"15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450\": container with ID starting with 15014b142a5486b120e84d4eba78db3b35fed1fb622e25d4bb7cfc931a360450 not found: ID does not exist"
Jul 28 17:38:17.114747 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.114722    2559 scope.go:117] "RemoveContainer" containerID="2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"
Jul 28 17:38:17.122239 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.122225    2559 scope.go:117] "RemoveContainer" containerID="bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"
Jul 28 17:38:17.123928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.123911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:38:17.129828 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.129809    2559 scope.go:117] "RemoveContainer" containerID="6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"
Jul 28 17:38:17.130523 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.130502    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf"]
Jul 28 17:38:17.137273 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.137257    2559 scope.go:117] "RemoveContainer" containerID="2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"
Jul 28 17:38:17.137503 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.137486    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1\": container with ID starting with 2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1 not found: ID does not exist" containerID="2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"
Jul 28 17:38:17.137579 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.137507    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1"} err="failed to get container status \"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1\": rpc error: code = NotFound desc = could not find container \"2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1\": container with ID starting with 2f918fbbc3dba3f250adc9951b55cf05b4ff03fd3354f15b10f1a143343a88d1 not found: ID does not exist"
Jul 28 17:38:17.137579 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.137521    2559 scope.go:117] "RemoveContainer" containerID="bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"
Jul 28 17:38:17.137743 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.137724    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078\": container with ID starting with bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078 not found: ID does not exist" containerID="bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"
Jul 28 17:38:17.137782 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.137757    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078"} err="failed to get container status \"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078\": rpc error: code = NotFound desc = could not find container \"bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078\": container with ID starting with bc97603d838d471eac38e5451f49694493f7bdf2731863777417a56383ce4078 not found: ID does not exist"
Jul 28 17:38:17.137782 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.137772    2559 scope.go:117] "RemoveContainer" containerID="6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"
Jul 28 17:38:17.138028 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:38:17.138012    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc\": container with ID starting with 6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc not found: ID does not exist" containerID="6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"
Jul 28 17:38:17.138100 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.138035    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc"} err="failed to get container status \"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc\": rpc error: code = NotFound desc = could not find container \"6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc\": container with ID starting with 6d08616f61257ea3fb17388a079e264a49486560673535756e4cbd1a051c11fc not found: ID does not exist"
Jul 28 17:38:17.672452 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.672420    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" path="/var/lib/kubelet/pods/2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a/volumes"
Jul 28 17:38:17.672941 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:17.672926    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" path="/var/lib/kubelet/pods/6d6a1647-5e02-4040-8217-cf9443531a73/volumes"
Jul 28 17:38:22.229841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.229813    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230152    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="storage-initializer"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230163    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="storage-initializer"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230183    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230188    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230199    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="storage-initializer"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230205    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="storage-initializer"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230211    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230217    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230225    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container"
Jul 28 17:38:22.230229 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230231    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230238    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230243    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230288    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kserve-container"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230297    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230303    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1a3fac-29bd-4a82-873a-fdaec9e7ec0a" containerName="kube-rbac-proxy"
Jul 28 17:38:22.230541 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.230310    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6a1647-5e02-4040-8217-cf9443531a73" containerName="kserve-container"
Jul 28 17:38:22.233322 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.233306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.235929 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235900    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:38:22.236049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235926    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-82c139-predictor-serving-cert\""
Jul 28 17:38:22.236049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235967    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:38:22.236049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235900    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:38:22.236049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:38:22.236049 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.235969    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-82c139-kube-rbac-proxy-sar-config\""
Jul 28 17:38:22.236577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.236561    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:38:22.250745 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.250723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:38:22.309586 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.309561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.309715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.309594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdnv\" (UniqueName: \"kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.309715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.309629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.309715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.309663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.309715 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.309690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.410574 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.410531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.410574 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.410590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.410806 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.410621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.410806 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.410658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.410806 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.410679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdnv\" (UniqueName: \"kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.411131 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.411109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.411414 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.411394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.411494 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.411448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.412971 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.412955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.420293 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.420270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdnv\" (UniqueName: \"kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv\") pod \"isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") " pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.543927 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.543850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:22.677491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:22.677462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:38:22.680010 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:38:22.679987    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853c2840_8595_4f84_bc5a_393d5da4a4e3.slice/crio-65b59bec628158ac9830ed61591964b0ffafc6c1106dfb100021a49dd63ebd6d WatchSource:0}: Error finding container 65b59bec628158ac9830ed61591964b0ffafc6c1106dfb100021a49dd63ebd6d: Status 404 returned error can't find the container with id 65b59bec628158ac9830ed61591964b0ffafc6c1106dfb100021a49dd63ebd6d
Jul 28 17:38:23.123029 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:23.122999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerStarted","Data":"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"}
Jul 28 17:38:23.123212 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:23.123053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerStarted","Data":"65b59bec628158ac9830ed61591964b0ffafc6c1106dfb100021a49dd63ebd6d"}
Jul 28 17:38:27.142117 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:27.142088    2559 generic.go:356] "Generic (PLEG): container finished" podID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerID="f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b" exitCode=0
Jul 28 17:38:27.142117 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:27.142120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerDied","Data":"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"}
Jul 28 17:38:28.148977 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:28.148944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerStarted","Data":"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"}
Jul 28 17:38:28.149483 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:28.149002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerStarted","Data":"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"}
Jul 28 17:38:28.149483 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:28.149232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:28.168335 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:28.168292    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podStartSLOduration=2.592126659 podStartE2EDuration="6.168279546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.576152887s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:28.167392976 +0000 UTC m=+1747.049800094" watchObservedRunningTime="2026-07-28 17:38:28.168279546 +0000 UTC m=+1747.050686660"
Jul 28 17:38:29.155028 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:29.154999    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:29.156219 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:29.156186    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:38:30.159861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:30.159826    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:38:35.164492 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:35.164466    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:38:35.164962 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:35.164936    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:38:45.165394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:45.165351    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:38:55.165862 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:38:55.165825    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:39:05.165301 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:05.165262    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:39:15.165168 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:15.165072    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:39:21.654581 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:21.654550    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:39:21.660338 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:21.660316    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:39:25.165837 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:25.165798    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:39:35.165191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:35.165154    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 17:39:45.165610 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:39:45.165582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:40:07.533977 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.533943    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:07.537246 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.537231    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.540365 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.540338    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-82c139-predictor-serving-cert\""
Jul 28 17:40:07.540491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.540367    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-82c139-dockercfg-595wc\""
Jul 28 17:40:07.540491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.540446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-82c139\""
Jul 28 17:40:07.540970 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.540954    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-82c139-kube-rbac-proxy-sar-config\""
Jul 28 17:40:07.554865 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.554842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:07.675198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.675172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.675382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.675207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.675382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.675249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfk4\" (UniqueName: \"kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.675382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.675299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.675382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.675335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776510 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776510 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776746 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfk4\" (UniqueName: \"kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776746 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776746 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.776953 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.776933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.777327 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.777305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.777389 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.777344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.779099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.779077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.787188 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.787137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfk4\" (UniqueName: \"kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4\") pod \"isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") " pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.847937 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.847914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:07.998491 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:07.998459    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:08.002632 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:40:08.002589    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc8dbd3_51e8_42e8_b1d1_f1b8841be411.slice/crio-1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb WatchSource:0}: Error finding container 1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb: Status 404 returned error can't find the container with id 1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb
Jul 28 17:40:08.597978 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:08.597941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerStarted","Data":"2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"}
Jul 28 17:40:08.598359 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:08.598018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerStarted","Data":"1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb"}
Jul 28 17:40:13.621137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:13.621111    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/0.log"
Jul 28 17:40:13.621522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:13.621150    2559 generic.go:356] "Generic (PLEG): container finished" podID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerID="2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30" exitCode=1
Jul 28 17:40:13.621522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:13.621227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerDied","Data":"2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"}
Jul 28 17:40:14.627685 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:14.627657    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/0.log"
Jul 28 17:40:14.628079 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:14.627756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerStarted","Data":"41cceee4e968674f16571f6313feddcfd5211fd84d3aee60e179571067dbfe2a"}
Jul 28 17:40:17.643800 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.643773    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/1.log"
Jul 28 17:40:17.644236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.644197    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/0.log"
Jul 28 17:40:17.644277 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.644229    2559 generic.go:356] "Generic (PLEG): container finished" podID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerID="41cceee4e968674f16571f6313feddcfd5211fd84d3aee60e179571067dbfe2a" exitCode=1
Jul 28 17:40:17.644327 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.644305    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerDied","Data":"41cceee4e968674f16571f6313feddcfd5211fd84d3aee60e179571067dbfe2a"}
Jul 28 17:40:17.644409 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.644398    2559 scope.go:117] "RemoveContainer" containerID="2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"
Jul 28 17:40:17.644765 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.644750    2559 scope.go:117] "RemoveContainer" containerID="2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"
Jul 28 17:40:17.655528 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:17.655505    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_kserve-ci-e2e-test(afc8dbd3-51e8-42e8-b1d1-f1b8841be411)\"" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411"
Jul 28 17:40:17.655718 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:17.655689    2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_kserve-ci-e2e-test_afc8dbd3-51e8-42e8-b1d1-f1b8841be411_0 in pod sandbox 1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb from index: no such id: '2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30'" containerID="2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"
Jul 28 17:40:17.655785 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:17.655729    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_kserve-ci-e2e-test_afc8dbd3-51e8-42e8-b1d1-f1b8841be411_0 in pod sandbox 1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb from index: no such id: '2870273cfaf429415446f0ef4d176289308c71f2605c3d8f59be414cfcc93c30'"
Jul 28 17:40:18.650252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:18.650224    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/1.log"
Jul 28 17:40:25.544052 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.544019    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:25.606928 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.606898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:40:25.607212 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.607184    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container" containerID="cri-o://55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650" gracePeriod=30
Jul 28 17:40:25.607353 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.607189    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kube-rbac-proxy" containerID="cri-o://0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1" gracePeriod=30
Jul 28 17:40:25.727724 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.727702    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/1.log"
Jul 28 17:40:25.729781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.729763    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:25.815099 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815075    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls\") pod \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") "
Jul 28 17:40:25.815235 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815143    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcfk4\" (UniqueName: \"kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4\") pod \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") "
Jul 28 17:40:25.815235 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815187    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location\") pod \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") "
Jul 28 17:40:25.815235 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert\") pod \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") "
Jul 28 17:40:25.815379 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config\") pod \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\" (UID: \"afc8dbd3-51e8-42e8-b1d1-f1b8841be411\") "
Jul 28 17:40:25.815442 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afc8dbd3-51e8-42e8-b1d1-f1b8841be411" (UID: "afc8dbd3-51e8-42e8-b1d1-f1b8841be411"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:25.815553 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815532    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:25.815650 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815625    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afc8dbd3-51e8-42e8-b1d1-f1b8841be411" (UID: "afc8dbd3-51e8-42e8-b1d1-f1b8841be411"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:25.815703 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.815641    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afc8dbd3-51e8-42e8-b1d1-f1b8841be411" (UID: "afc8dbd3-51e8-42e8-b1d1-f1b8841be411"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:25.817137 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.817114    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4" (OuterVolumeSpecName: "kube-api-access-xcfk4") pod "afc8dbd3-51e8-42e8-b1d1-f1b8841be411" (UID: "afc8dbd3-51e8-42e8-b1d1-f1b8841be411"). InnerVolumeSpecName "kube-api-access-xcfk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:25.817191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.817176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afc8dbd3-51e8-42e8-b1d1-f1b8841be411" (UID: "afc8dbd3-51e8-42e8-b1d1-f1b8841be411"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:25.916172 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.916146    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfk4\" (UniqueName: \"kubernetes.io/projected/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-api-access-xcfk4\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:25.916172 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.916168    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:25.916172 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.916178    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:25.916360 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:25.916186    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc8dbd3-51e8-42e8-b1d1-f1b8841be411-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:26.686989 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.686949    2559 generic.go:356] "Generic (PLEG): container finished" podID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerID="0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1" exitCode=2
Jul 28 17:40:26.687424 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.687025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerDied","Data":"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"}
Jul 28 17:40:26.688289 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.688274    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_afc8dbd3-51e8-42e8-b1d1-f1b8841be411/storage-initializer/1.log"
Jul 28 17:40:26.688357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.688328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf" event={"ID":"afc8dbd3-51e8-42e8-b1d1-f1b8841be411","Type":"ContainerDied","Data":"1f71d3f1cea678c3d6f59ccc3c4e52eee61b10ff5bb807fa402ef319cef6dabb"}
Jul 28 17:40:26.688395 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.688368    2559 scope.go:117] "RemoveContainer" containerID="41cceee4e968674f16571f6313feddcfd5211fd84d3aee60e179571067dbfe2a"
Jul 28 17:40:26.688395 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.688373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"
Jul 28 17:40:26.724104 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.724075    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:26.728459 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:26.728437    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf"]
Jul 28 17:40:27.672652 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:27.672623    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" path="/var/lib/kubelet/pods/afc8dbd3-51e8-42e8-b1d1-f1b8841be411/volumes"
Jul 28 17:40:30.056310 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.056289    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:40:30.153357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153284    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert\") pod \"853c2840-8595-4f84-bc5a-393d5da4a4e3\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") "
Jul 28 17:40:30.153357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153320    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwdnv\" (UniqueName: \"kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv\") pod \"853c2840-8595-4f84-bc5a-393d5da4a4e3\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") "
Jul 28 17:40:30.153357 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153342    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls\") pod \"853c2840-8595-4f84-bc5a-393d5da4a4e3\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") "
Jul 28 17:40:30.153577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config\") pod \"853c2840-8595-4f84-bc5a-393d5da4a4e3\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") "
Jul 28 17:40:30.153577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153512    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location\") pod \"853c2840-8595-4f84-bc5a-393d5da4a4e3\" (UID: \"853c2840-8595-4f84-bc5a-393d5da4a4e3\") "
Jul 28 17:40:30.153789 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "853c2840-8595-4f84-bc5a-393d5da4a4e3" (UID: "853c2840-8595-4f84-bc5a-393d5da4a4e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:30.153933 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153793    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "853c2840-8595-4f84-bc5a-393d5da4a4e3" (UID: "853c2840-8595-4f84-bc5a-393d5da4a4e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:30.153933 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.153824    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "853c2840-8595-4f84-bc5a-393d5da4a4e3" (UID: "853c2840-8595-4f84-bc5a-393d5da4a4e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:30.155479 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.155457    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "853c2840-8595-4f84-bc5a-393d5da4a4e3" (UID: "853c2840-8595-4f84-bc5a-393d5da4a4e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:30.155560 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.155541    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv" (OuterVolumeSpecName: "kube-api-access-wwdnv") pod "853c2840-8595-4f84-bc5a-393d5da4a4e3" (UID: "853c2840-8595-4f84-bc5a-393d5da4a4e3"). InnerVolumeSpecName "kube-api-access-wwdnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:30.254707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.254681    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:30.254707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.254701    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853c2840-8595-4f84-bc5a-393d5da4a4e3-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:30.254707 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.254711    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853c2840-8595-4f84-bc5a-393d5da4a4e3-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:30.254921 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.254720    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwdnv\" (UniqueName: \"kubernetes.io/projected/853c2840-8595-4f84-bc5a-393d5da4a4e3-kube-api-access-wwdnv\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:30.254921 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.254728    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/853c2840-8595-4f84-bc5a-393d5da4a4e3-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:30.709739 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.709703    2559 generic.go:356] "Generic (PLEG): container finished" podID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerID="55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650" exitCode=0
Jul 28 17:40:30.709949 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.709773    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"
Jul 28 17:40:30.709949 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.709802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerDied","Data":"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"}
Jul 28 17:40:30.709949 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.709892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl" event={"ID":"853c2840-8595-4f84-bc5a-393d5da4a4e3","Type":"ContainerDied","Data":"65b59bec628158ac9830ed61591964b0ffafc6c1106dfb100021a49dd63ebd6d"}
Jul 28 17:40:30.709949 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.709899    2559 scope.go:117] "RemoveContainer" containerID="0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"
Jul 28 17:40:30.719011 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.718991    2559 scope.go:117] "RemoveContainer" containerID="55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"
Jul 28 17:40:30.726696 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.726675    2559 scope.go:117] "RemoveContainer" containerID="f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"
Jul 28 17:40:30.734490 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.734455    2559 scope.go:117] "RemoveContainer" containerID="0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"
Jul 28 17:40:30.734718 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:30.734695    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1\": container with ID starting with 0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1 not found: ID does not exist" containerID="0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"
Jul 28 17:40:30.734787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.734726    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1"} err="failed to get container status \"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1\": rpc error: code = NotFound desc = could not find container \"0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1\": container with ID starting with 0341bd85c5855d3f3339312c931d7e872dc640f9a5e96ffd202a7c1c496d90a1 not found: ID does not exist"
Jul 28 17:40:30.734787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.734744    2559 scope.go:117] "RemoveContainer" containerID="55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"
Jul 28 17:40:30.734999 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:30.734982    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650\": container with ID starting with 55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650 not found: ID does not exist" containerID="55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"
Jul 28 17:40:30.735066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.735003    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650"} err="failed to get container status \"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650\": rpc error: code = NotFound desc = could not find container \"55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650\": container with ID starting with 55e7a4ab2dc5bb88b9d271d55ea607499d7ddc660e4486d6e8fed4d190588650 not found: ID does not exist"
Jul 28 17:40:30.735066 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.735018    2559 scope.go:117] "RemoveContainer" containerID="f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"
Jul 28 17:40:30.735258 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:30.735240    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b\": container with ID starting with f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b not found: ID does not exist" containerID="f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"
Jul 28 17:40:30.735313 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.735262    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b"} err="failed to get container status \"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b\": rpc error: code = NotFound desc = could not find container \"f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b\": container with ID starting with f7cdaf58d3a5193062ddfa9e6e4d4284f78064b9371cb73032a703a5b3a5cb6b not found: ID does not exist"
Jul 28 17:40:30.736160 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.736140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:40:30.741195 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.741176    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl"]
Jul 28 17:40:30.772439 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772413    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:30.772740 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772728    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container"
Jul 28 17:40:30.772784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772742    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container"
Jul 28 17:40:30.772784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772761    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.772784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772767    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.772784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772774    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="storage-initializer"
Jul 28 17:40:30.772784 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772780    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="storage-initializer"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772792    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kube-rbac-proxy"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772798    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kube-rbac-proxy"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772805    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772811    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772863    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772887    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kserve-container"
Jul 28 17:40:30.772951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.772895    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" containerName="kube-rbac-proxy"
Jul 28 17:40:30.773159 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.773000    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc8dbd3-51e8-42e8-b1d1-f1b8841be411" containerName="storage-initializer"
Jul 28 17:40:30.777521 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.777506    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.779753 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.779728    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:40:30.779862 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.779729    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fbe44c-predictor-serving-cert\""
Jul 28 17:40:30.779862 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.779839    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-fbe44c\""
Jul 28 17:40:30.780015 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.779861    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-fbe44c-dockercfg-s86lp\""
Jul 28 17:40:30.780015 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.779914    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:40:30.780111 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.780098    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fbe44c-kube-rbac-proxy-sar-config\""
Jul 28 17:40:30.780177 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.780165    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:40:30.787951 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.787929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:30.859170 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.859143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.859170 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.859172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.859381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.859191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.859381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.859250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.859381 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.859348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzd7z\" (UniqueName: \"kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960340 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960630 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960630 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzd7z\" (UniqueName: \"kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.960735 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.960671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.961174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.961154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.961276 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.961154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.962713 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.962692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:30.970199 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:30.970178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzd7z\" (UniqueName: \"kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z\") pod \"isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") " pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:31.088174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:31.088141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:31.226259 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:31.226219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:31.228371 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:40:31.228345    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5d9afe_54c7_4db7_98f8_eb082654e3aa.slice/crio-db95856236a34f78199426e00e486eab5e6f6609acb6d8c29dd1ce5872c39215 WatchSource:0}: Error finding container db95856236a34f78199426e00e486eab5e6f6609acb6d8c29dd1ce5872c39215: Status 404 returned error can't find the container with id db95856236a34f78199426e00e486eab5e6f6609acb6d8c29dd1ce5872c39215
Jul 28 17:40:31.674048 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:31.674016    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="853c2840-8595-4f84-bc5a-393d5da4a4e3" path="/var/lib/kubelet/pods/853c2840-8595-4f84-bc5a-393d5da4a4e3/volumes"
Jul 28 17:40:31.716334 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:31.716301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerStarted","Data":"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"}
Jul 28 17:40:31.716506 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:31.716357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerStarted","Data":"db95856236a34f78199426e00e486eab5e6f6609acb6d8c29dd1ce5872c39215"}
Jul 28 17:40:34.732557 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:34.732530    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/0.log"
Jul 28 17:40:34.732942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:34.732569    2559 generic.go:356] "Generic (PLEG): container finished" podID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerID="a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf" exitCode=1
Jul 28 17:40:34.732942 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:34.732604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerDied","Data":"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"}
Jul 28 17:40:35.739643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:35.739615    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/0.log"
Jul 28 17:40:35.740072 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:35.739715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerStarted","Data":"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"}
Jul 28 17:40:35.740138 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:35.740081    2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" secret="" err="secret \"fail-s3-sa-fbe44c-dockercfg-s86lp\" not found"
Jul 28 17:40:35.779245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:35.779211    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:35.804645 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:35.804619    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-serving-cert: secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:35.804792 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:35.804692    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls podName:fd5d9afe-54c7-4db7-98f8-eb082654e3aa nodeName:}" failed. No retries permitted until 2026-07-28 17:40:36.304672994 +0000 UTC m=+1875.187080087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls") pod "isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa") : secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:36.308484 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:36.308451    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-serving-cert: secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:36.308644 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:36.308527    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls podName:fd5d9afe-54c7-4db7-98f8-eb082654e3aa nodeName:}" failed. No retries permitted until 2026-07-28 17:40:37.308512416 +0000 UTC m=+1876.190919514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls") pod "isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa") : secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:36.745029 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:36.744991    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer" containerID="cri-o://ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b" gracePeriod=30
Jul 28 17:40:37.317901 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:37.317852    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-serving-cert: secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:37.318101 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:37.317941    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls podName:fd5d9afe-54c7-4db7-98f8-eb082654e3aa nodeName:}" failed. No retries permitted until 2026-07-28 17:40:39.317922314 +0000 UTC m=+1878.200329409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls") pod "isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa") : secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found
Jul 28 17:40:39.293384 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.293360    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/1.log"
Jul 28 17:40:39.293769 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.293755    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/0.log"
Jul 28 17:40:39.296048 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.296033    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:39.334865 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.334841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location\") pod \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") "
Jul 28 17:40:39.335006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.334921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzd7z\" (UniqueName: \"kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z\") pod \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") "
Jul 28 17:40:39.335006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.334966    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert\") pod \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") "
Jul 28 17:40:39.335006 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.334989    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls\") pod \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") "
Jul 28 17:40:39.335174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.335017    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config\") pod \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\" (UID: \"fd5d9afe-54c7-4db7-98f8-eb082654e3aa\") "
Jul 28 17:40:39.335174 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.335126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd5d9afe-54c7-4db7-98f8-eb082654e3aa" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:39.335304 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.335286    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:39.335358 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.335336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd5d9afe-54c7-4db7-98f8-eb082654e3aa" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:39.335407 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.335390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd5d9afe-54c7-4db7-98f8-eb082654e3aa" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:39.337022 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.337002    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd5d9afe-54c7-4db7-98f8-eb082654e3aa" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:39.337121 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.337103    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z" (OuterVolumeSpecName: "kube-api-access-kzd7z") pod "fd5d9afe-54c7-4db7-98f8-eb082654e3aa" (UID: "fd5d9afe-54c7-4db7-98f8-eb082654e3aa"). InnerVolumeSpecName "kube-api-access-kzd7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:39.435787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.435719    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:39.435787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.435744    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:39.435787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.435753    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:39.435787 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.435762    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzd7z\" (UniqueName: \"kubernetes.io/projected/fd5d9afe-54c7-4db7-98f8-eb082654e3aa-kube-api-access-kzd7z\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:40:39.760197 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/1.log"
Jul 28 17:40:39.760522 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760507    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn_fd5d9afe-54c7-4db7-98f8-eb082654e3aa/storage-initializer/0.log"
Jul 28 17:40:39.760582 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760538    2559 generic.go:356] "Generic (PLEG): container finished" podID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerID="ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b" exitCode=1
Jul 28 17:40:39.760632 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760601    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"
Jul 28 17:40:39.760667 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerDied","Data":"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"}
Jul 28 17:40:39.760711 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn" event={"ID":"fd5d9afe-54c7-4db7-98f8-eb082654e3aa","Type":"ContainerDied","Data":"db95856236a34f78199426e00e486eab5e6f6609acb6d8c29dd1ce5872c39215"}
Jul 28 17:40:39.760752 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.760711    2559 scope.go:117] "RemoveContainer" containerID="ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"
Jul 28 17:40:39.769170 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.769152    2559 scope.go:117] "RemoveContainer" containerID="a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"
Jul 28 17:40:39.776509 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.776494    2559 scope.go:117] "RemoveContainer" containerID="ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"
Jul 28 17:40:39.776743 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:39.776728    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b\": container with ID starting with ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b not found: ID does not exist" containerID="ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"
Jul 28 17:40:39.776789 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.776758    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b"} err="failed to get container status \"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b\": rpc error: code = NotFound desc = could not find container \"ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b\": container with ID starting with ad62c0e81bb7862b8c5573d6b814e97f1cd3bceb57249bec080ed6efa313996b not found: ID does not exist"
Jul 28 17:40:39.776789 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.776773    2559 scope.go:117] "RemoveContainer" containerID="a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"
Jul 28 17:40:39.777101 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:40:39.777086    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf\": container with ID starting with a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf not found: ID does not exist" containerID="a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"
Jul 28 17:40:39.777159 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.777107    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf"} err="failed to get container status \"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf\": rpc error: code = NotFound desc = could not find container \"a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf\": container with ID starting with a8655d8a228b0b5a29ec164ef96c5999f44379bbe59e37b6cef8a2db875265cf not found: ID does not exist"
Jul 28 17:40:39.797242 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.797217    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:39.802393 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:39.802366    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn"]
Jul 28 17:40:41.095983 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.095942    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:40:41.096386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.096386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096327    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.096386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096339    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.096386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096344    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.096537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096401    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.096537 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.096412    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" containerName="storage-initializer"
Jul 28 17:40:41.102036 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.102006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.104818 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.104794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:41.105001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.104833    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fb75d-predictor-serving-cert\""
Jul 28 17:40:41.105001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.104834    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:40:41.105001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.104910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fb75d-kube-rbac-proxy-sar-config\""
Jul 28 17:40:41.105591 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.105572    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:40:41.105699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.105616    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:40:41.105699 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.105615    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:40:41.112966 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.112946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:40:41.148615 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.148590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.148739 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.148627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.148739 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.148646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.148818 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.148751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.148818 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.148786    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rp5\" (UniqueName: \"kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249198 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249436 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249436 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rp5\" (UniqueName: \"kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249436 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249649 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249624    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249952 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.249952 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.249940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.251705 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.251680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.260450 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.260431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rp5\" (UniqueName: \"kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5\") pod \"raw-sklearn-fb75d-predictor-768dc4cffb-4phk2\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") " pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.413606 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.413527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:41.553149 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.553128    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:40:41.555059 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:40:41.555033    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617cd576_20c3_425f_9148_130b16092fe1.slice/crio-3abe9412e4e1bb2cd9ca87894afae71fed95488d495b8449df978d0411a9a240 WatchSource:0}: Error finding container 3abe9412e4e1bb2cd9ca87894afae71fed95488d495b8449df978d0411a9a240: Status 404 returned error can't find the container with id 3abe9412e4e1bb2cd9ca87894afae71fed95488d495b8449df978d0411a9a240
Jul 28 17:40:41.674561 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.674480    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5d9afe-54c7-4db7-98f8-eb082654e3aa" path="/var/lib/kubelet/pods/fd5d9afe-54c7-4db7-98f8-eb082654e3aa/volumes"
Jul 28 17:40:41.772216 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.772178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerStarted","Data":"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"}
Jul 28 17:40:41.772543 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:41.772234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerStarted","Data":"3abe9412e4e1bb2cd9ca87894afae71fed95488d495b8449df978d0411a9a240"}
Jul 28 17:40:45.791803 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:45.791768    2559 generic.go:356] "Generic (PLEG): container finished" podID="617cd576-20c3-425f-9148-130b16092fe1" containerID="679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc" exitCode=0
Jul 28 17:40:45.792277 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:45.791821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerDied","Data":"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"}
Jul 28 17:40:46.798933 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:46.798897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerStarted","Data":"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"}
Jul 28 17:40:46.799309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:46.798984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerStarted","Data":"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"}
Jul 28 17:40:46.799309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:46.799179    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:46.820453 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:46.820397    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podStartSLOduration=1.814356343 podStartE2EDuration="5.820378515s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.006022172s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:46.818039638 +0000 UTC m=+1885.700446755" watchObservedRunningTime="2026-07-28 17:40:46.820378515 +0000 UTC m=+1885.702785631"
Jul 28 17:40:47.804183 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:47.804156    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:47.805434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:47.805413    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:40:48.809450 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:48.809406    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:40:53.814238 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:53.814211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:40:53.814785 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:40:53.814757    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:03.815640 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:03.815600    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:13.815434 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:13.815396    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:23.814779 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:23.814745    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:33.815034 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:33.814995    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:43.815384 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:43.815346    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:41:53.814626 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:41:53.814577    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:42:03.816206 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:03.816173    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:42:11.191165 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:11.191115    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:42:11.191621 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:11.191489    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" containerID="cri-o://1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83" gracePeriod=30
Jul 28 17:42:11.191695 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:11.191664    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kube-rbac-proxy" containerID="cri-o://ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44" gracePeriod=30
Jul 28 17:42:12.184427 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:12.184396    2559 generic.go:356] "Generic (PLEG): container finished" podID="617cd576-20c3-425f-9148-130b16092fe1" containerID="ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44" exitCode=2
Jul 28 17:42:12.184601 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:12.184436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerDied","Data":"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"}
Jul 28 17:42:13.810307 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:13.810267    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 28 17:42:13.814649 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:13.814619    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 17:42:15.841853 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.841827    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:42:15.934314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934230    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert\") pod \"617cd576-20c3-425f-9148-130b16092fe1\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") "
Jul 28 17:42:15.934314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934270    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls\") pod \"617cd576-20c3-425f-9148-130b16092fe1\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") "
Jul 28 17:42:15.934314 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location\") pod \"617cd576-20c3-425f-9148-130b16092fe1\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") "
Jul 28 17:42:15.934542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934381    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78rp5\" (UniqueName: \"kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5\") pod \"617cd576-20c3-425f-9148-130b16092fe1\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") "
Jul 28 17:42:15.934542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934498    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config\") pod \"617cd576-20c3-425f-9148-130b16092fe1\" (UID: \"617cd576-20c3-425f-9148-130b16092fe1\") "
Jul 28 17:42:15.934680 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934653    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "617cd576-20c3-425f-9148-130b16092fe1" (UID: "617cd576-20c3-425f-9148-130b16092fe1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:15.934742 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934655    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "617cd576-20c3-425f-9148-130b16092fe1" (UID: "617cd576-20c3-425f-9148-130b16092fe1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:15.934831 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934813    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cd576-20c3-425f-9148-130b16092fe1-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.934926 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934836    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:42:15.934926 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.934860    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "617cd576-20c3-425f-9148-130b16092fe1" (UID: "617cd576-20c3-425f-9148-130b16092fe1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:15.936419 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.936395    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "617cd576-20c3-425f-9148-130b16092fe1" (UID: "617cd576-20c3-425f-9148-130b16092fe1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:15.936501 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:15.936433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5" (OuterVolumeSpecName: "kube-api-access-78rp5") pod "617cd576-20c3-425f-9148-130b16092fe1" (UID: "617cd576-20c3-425f-9148-130b16092fe1"). InnerVolumeSpecName "kube-api-access-78rp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:16.035860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.035826    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rp5\" (UniqueName: \"kubernetes.io/projected/617cd576-20c3-425f-9148-130b16092fe1-kube-api-access-78rp5\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:42:16.035860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.035851    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cd576-20c3-425f-9148-130b16092fe1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:42:16.035860 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.035861    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cd576-20c3-425f-9148-130b16092fe1-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:42:16.204390 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.204306    2559 generic.go:356] "Generic (PLEG): container finished" podID="617cd576-20c3-425f-9148-130b16092fe1" containerID="1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83" exitCode=0
Jul 28 17:42:16.204390 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.204369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"
Jul 28 17:42:16.204604 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.204384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerDied","Data":"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"}
Jul 28 17:42:16.204604 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.204444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2" event={"ID":"617cd576-20c3-425f-9148-130b16092fe1","Type":"ContainerDied","Data":"3abe9412e4e1bb2cd9ca87894afae71fed95488d495b8449df978d0411a9a240"}
Jul 28 17:42:16.204604 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.204476    2559 scope.go:117] "RemoveContainer" containerID="ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"
Jul 28 17:42:16.218134 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.215844    2559 scope.go:117] "RemoveContainer" containerID="1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"
Jul 28 17:42:16.226397 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.226376    2559 scope.go:117] "RemoveContainer" containerID="679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"
Jul 28 17:42:16.234953 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.234850    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:42:16.235026 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.234971    2559 scope.go:117] "RemoveContainer" containerID="ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"
Jul 28 17:42:16.235246 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:42:16.235227    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44\": container with ID starting with ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44 not found: ID does not exist" containerID="ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"
Jul 28 17:42:16.235315 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.235254    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44"} err="failed to get container status \"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44\": rpc error: code = NotFound desc = could not find container \"ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44\": container with ID starting with ded5bf04f8f67c3ba52524eb4b5eaef12d895c73765bd278ba10fba3b5b83e44 not found: ID does not exist"
Jul 28 17:42:16.235315 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.235271    2559 scope.go:117] "RemoveContainer" containerID="1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"
Jul 28 17:42:16.235507 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:42:16.235487    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83\": container with ID starting with 1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83 not found: ID does not exist" containerID="1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"
Jul 28 17:42:16.235566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.235511    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83"} err="failed to get container status \"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83\": rpc error: code = NotFound desc = could not find container \"1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83\": container with ID starting with 1902c00708413fdedb000f58f5b0cb47c7f5c151a3faaf6a696a1787a6259d83 not found: ID does not exist"
Jul 28 17:42:16.235566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.235524    2559 scope.go:117] "RemoveContainer" containerID="679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"
Jul 28 17:42:16.235724 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:42:16.235706    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc\": container with ID starting with 679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc not found: ID does not exist" containerID="679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"
Jul 28 17:42:16.235781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.235728    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc"} err="failed to get container status \"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc\": rpc error: code = NotFound desc = could not find container \"679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc\": container with ID starting with 679743c367efd1b65d92798bff5d8be9e4f4f5422c9c78dafaca98bf235617bc not found: ID does not exist"
Jul 28 17:42:16.236643 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.236621    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2"]
Jul 28 17:42:16.377113 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377079    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:42:16.377449 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377436    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="storage-initializer"
Jul 28 17:42:16.377449 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377450    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="storage-initializer"
Jul 28 17:42:16.377534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377467    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container"
Jul 28 17:42:16.377534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377473    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container"
Jul 28 17:42:16.377534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377489    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kube-rbac-proxy"
Jul 28 17:42:16.377534 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377495    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kube-rbac-proxy"
Jul 28 17:42:16.377659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377552    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kube-rbac-proxy"
Jul 28 17:42:16.377659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.377564    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="617cd576-20c3-425f-9148-130b16092fe1" containerName="kserve-container"
Jul 28 17:42:16.382181 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.382163    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.384433 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384410    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:42:16.384568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:42:16.384568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384501    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-79274-kube-rbac-proxy-sar-config\""
Jul 28 17:42:16.384568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:42:16.384740 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384726    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-79274-predictor-serving-cert\""
Jul 28 17:42:16.384906 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.384892    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:42:16.385092 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.385079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-klwxk\""
Jul 28 17:42:16.389615 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.389591    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:42:16.539568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.539481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.539568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.539517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgvp\" (UniqueName: \"kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.539568 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.539558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.539781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.539576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.539781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.539624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640382 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640617 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640617 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640724 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgvp\" (UniqueName: \"kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.640955 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.640933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.641179 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.641160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.641255 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.641232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.642779 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.642754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.649360 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.649330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgvp\" (UniqueName: \"kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp\") pod \"raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.693993 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.693965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:16.827473 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:16.827448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:42:16.829739 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:42:16.829713    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73491271_b88f_45f1_9bf9_6ef9ef9ac34d.slice/crio-14acc87481ac5bff3fe051dd89aa37c5455aec84d1164f95f11a8d4cd76dcc6c WatchSource:0}: Error finding container 14acc87481ac5bff3fe051dd89aa37c5455aec84d1164f95f11a8d4cd76dcc6c: Status 404 returned error can't find the container with id 14acc87481ac5bff3fe051dd89aa37c5455aec84d1164f95f11a8d4cd76dcc6c
Jul 28 17:42:17.211374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:17.211278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerStarted","Data":"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"}
Jul 28 17:42:17.211374 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:17.211354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerStarted","Data":"14acc87481ac5bff3fe051dd89aa37c5455aec84d1164f95f11a8d4cd76dcc6c"}
Jul 28 17:42:17.673517 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:17.673483    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617cd576-20c3-425f-9148-130b16092fe1" path="/var/lib/kubelet/pods/617cd576-20c3-425f-9148-130b16092fe1/volumes"
Jul 28 17:42:21.232097 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:21.232009    2559 generic.go:356] "Generic (PLEG): container finished" podID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerID="82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17" exitCode=0
Jul 28 17:42:21.232565 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:21.232086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerDied","Data":"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"}
Jul 28 17:42:22.238493 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:22.238455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerStarted","Data":"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"}
Jul 28 17:42:22.238995 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:22.238529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerStarted","Data":"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"}
Jul 28 17:42:22.238995 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:22.238745    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:22.262243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:22.262197    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podStartSLOduration=2.341410122 podStartE2EDuration="6.262182948s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920772826s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:22.261397387 +0000 UTC m=+1981.143804502" watchObservedRunningTime="2026-07-28 17:42:22.262182948 +0000 UTC m=+1981.144590063"
Jul 28 17:42:23.243632 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:23.243601    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:23.244910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:23.244886    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:42:24.248463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:24.248425    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:42:29.253793 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:29.253759    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:42:29.254262 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:29.254230    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:42:39.254955 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:39.254912    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:42:49.255082 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:49.255033    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:42:59.254638 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:42:59.254595    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:43:09.255083 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:09.255038    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:43:19.254753 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:19.254711    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:43:29.254852 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:29.254810    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:43:37.672622 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:37.672594    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:43:46.542178 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:46.542098    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:43:46.542617 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:46.542425    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kube-rbac-proxy" containerID="cri-o://8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc" gracePeriod=30
Jul 28 17:43:46.542696 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:46.542635    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" containerID="cri-o://785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d" gracePeriod=30
Jul 28 17:43:47.625592 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:47.625558    2559 generic.go:356] "Generic (PLEG): container finished" podID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerID="8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc" exitCode=2
Jul 28 17:43:47.626014 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:47.625635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerDied","Data":"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"}
Jul 28 17:43:47.667658 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:47.667619    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 17:43:49.248837 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:49.248792    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 28 17:43:51.201334 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.201308    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:43:51.329440 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls\") pod \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") "
Jul 28 17:43:51.329629 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config\") pod \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") "
Jul 28 17:43:51.329629 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location\") pod \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") "
Jul 28 17:43:51.329629 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329535    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcgvp\" (UniqueName: \"kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp\") pod \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") "
Jul 28 17:43:51.329796 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329634    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert\") pod \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\" (UID: \"73491271-b88f-45f1-9bf9-6ef9ef9ac34d\") "
Jul 28 17:43:51.329858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329794    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73491271-b88f-45f1-9bf9-6ef9ef9ac34d" (UID: "73491271-b88f-45f1-9bf9-6ef9ef9ac34d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:51.329858 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329820    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73491271-b88f-45f1-9bf9-6ef9ef9ac34d" (UID: "73491271-b88f-45f1-9bf9-6ef9ef9ac34d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:51.330008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329946    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:43:51.330008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.329962    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kserve-provision-location\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:43:51.330106 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.330032    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73491271-b88f-45f1-9bf9-6ef9ef9ac34d" (UID: "73491271-b88f-45f1-9bf9-6ef9ef9ac34d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:51.331652 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.331629    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73491271-b88f-45f1-9bf9-6ef9ef9ac34d" (UID: "73491271-b88f-45f1-9bf9-6ef9ef9ac34d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:51.331762 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.331681    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp" (OuterVolumeSpecName: "kube-api-access-pcgvp") pod "73491271-b88f-45f1-9bf9-6ef9ef9ac34d" (UID: "73491271-b88f-45f1-9bf9-6ef9ef9ac34d"). InnerVolumeSpecName "kube-api-access-pcgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:51.431191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.431148    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-proxy-tls\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:43:51.431191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.431182    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcgvp\" (UniqueName: \"kubernetes.io/projected/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-kube-api-access-pcgvp\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:43:51.431191 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.431195    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73491271-b88f-45f1-9bf9-6ef9ef9ac34d-cabundle-cert\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:43:51.646526 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.646439    2559 generic.go:356] "Generic (PLEG): container finished" podID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerID="785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d" exitCode=0
Jul 28 17:43:51.646526 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.646501    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"
Jul 28 17:43:51.646770 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.646528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerDied","Data":"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"}
Jul 28 17:43:51.646770 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.646609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx" event={"ID":"73491271-b88f-45f1-9bf9-6ef9ef9ac34d","Type":"ContainerDied","Data":"14acc87481ac5bff3fe051dd89aa37c5455aec84d1164f95f11a8d4cd76dcc6c"}
Jul 28 17:43:51.646770 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.646648    2559 scope.go:117] "RemoveContainer" containerID="8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"
Jul 28 17:43:51.655692 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.655674    2559 scope.go:117] "RemoveContainer" containerID="785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"
Jul 28 17:43:51.662851 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.662832    2559 scope.go:117] "RemoveContainer" containerID="82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"
Jul 28 17:43:51.670205 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670185    2559 scope.go:117] "RemoveContainer" containerID="8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"
Jul 28 17:43:51.670457 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:43:51.670438    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc\": container with ID starting with 8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc not found: ID does not exist" containerID="8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"
Jul 28 17:43:51.670519 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670465    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc"} err="failed to get container status \"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc\": rpc error: code = NotFound desc = could not find container \"8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc\": container with ID starting with 8c11ca024064dbd0bfdad26a9ceace9fecbbd1fb72e0be28c680f15db8fe49cc not found: ID does not exist"
Jul 28 17:43:51.670519 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670483    2559 scope.go:117] "RemoveContainer" containerID="785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"
Jul 28 17:43:51.670725 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:43:51.670709    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d\": container with ID starting with 785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d not found: ID does not exist" containerID="785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"
Jul 28 17:43:51.670767 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670730    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d"} err="failed to get container status \"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d\": rpc error: code = NotFound desc = could not find container \"785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d\": container with ID starting with 785c26d125ecb471f1bc4ac4f460e2a1667864eb538070ad8b52d5d47b2d2d5d not found: ID does not exist"
Jul 28 17:43:51.670767 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670745    2559 scope.go:117] "RemoveContainer" containerID="82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"
Jul 28 17:43:51.670962 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:43:51.670944    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17\": container with ID starting with 82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17 not found: ID does not exist" containerID="82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"
Jul 28 17:43:51.671012 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.670969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17"} err="failed to get container status \"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17\": rpc error: code = NotFound desc = could not find container \"82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17\": container with ID starting with 82d22b596cdb544ba227c4070dfed21b9cd12cbb4951fa20329c20512e4e5b17 not found: ID does not exist"
Jul 28 17:43:51.674326 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.674308    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:43:51.678963 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:51.678942    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx"]
Jul 28 17:43:52.681851 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.681823    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spz9m/must-gather-pr486"]
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682191    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container"
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container"
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682210    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kube-rbac-proxy"
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682215    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kube-rbac-proxy"
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682225    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="storage-initializer"
Jul 28 17:43:52.682233 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682231    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="storage-initializer"
Jul 28 17:43:52.682542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682307    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kserve-container"
Jul 28 17:43:52.682542 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.682327    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" containerName="kube-rbac-proxy"
Jul 28 17:43:52.686811 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.686793    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.689114 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.689097    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spz9m\"/\"openshift-service-ca.crt\""
Jul 28 17:43:52.690028 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.690001    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-spz9m\"/\"default-dockercfg-g7nbk\""
Jul 28 17:43:52.690028 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.690005    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spz9m\"/\"kube-root-ca.crt\""
Jul 28 17:43:52.692750 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.692729    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spz9m/must-gather-pr486"]
Jul 28 17:43:52.740619 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.740592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxsh\" (UniqueName: \"kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.740759 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.740649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.841279 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.841250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.841394 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.841305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxsh\" (UniqueName: \"kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.841564 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.841546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:52.850807 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:52.850790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxsh\" (UniqueName: \"kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh\") pod \"must-gather-pr486\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") " pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:53.003341 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:53.003265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:43:53.134161 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:53.134136    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spz9m/must-gather-pr486"]
Jul 28 17:43:53.136533 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:43:53.136508    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93ce979f_ca54_4c37_9cb5_b9bc13213159.slice/crio-691fb96a2a25952c2070a844455854382d64bc64980d91e8b8b9c813d623feef WatchSource:0}: Error finding container 691fb96a2a25952c2070a844455854382d64bc64980d91e8b8b9c813d623feef: Status 404 returned error can't find the container with id 691fb96a2a25952c2070a844455854382d64bc64980d91e8b8b9c813d623feef
Jul 28 17:43:53.138309 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:53.138293    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:43:53.657536 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:53.657497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spz9m/must-gather-pr486" event={"ID":"93ce979f-ca54-4c37-9cb5-b9bc13213159","Type":"ContainerStarted","Data":"691fb96a2a25952c2070a844455854382d64bc64980d91e8b8b9c813d623feef"}
Jul 28 17:43:53.672232 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:53.672197    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73491271-b88f-45f1-9bf9-6ef9ef9ac34d" path="/var/lib/kubelet/pods/73491271-b88f-45f1-9bf9-6ef9ef9ac34d/volumes"
Jul 28 17:43:59.698243 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:59.698202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spz9m/must-gather-pr486" event={"ID":"93ce979f-ca54-4c37-9cb5-b9bc13213159","Type":"ContainerStarted","Data":"bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037"}
Jul 28 17:43:59.698659 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:59.698262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spz9m/must-gather-pr486" event={"ID":"93ce979f-ca54-4c37-9cb5-b9bc13213159","Type":"ContainerStarted","Data":"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"}
Jul 28 17:43:59.715750 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:43:59.715698    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spz9m/must-gather-pr486" podStartSLOduration=1.637553692 podStartE2EDuration="7.715684268s" totalImagesPullingTime="6.078130576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:59.714553581 +0000 UTC m=+2078.596960691" watchObservedRunningTime="2026-07-28 17:43:59.715684268 +0000 UTC m=+2078.598091424"
Jul 28 17:44:21.686415 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:21.686390    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:44:21.690759 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:21.690737    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:44:21.817582 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:21.817555    2559 generic.go:356] "Generic (PLEG): container finished" podID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerID="2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64" exitCode=0
Jul 28 17:44:21.817680 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:21.817625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spz9m/must-gather-pr486" event={"ID":"93ce979f-ca54-4c37-9cb5-b9bc13213159","Type":"ContainerDied","Data":"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"}
Jul 28 17:44:21.817983 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:21.817968    2559 scope.go:117] "RemoveContainer" containerID="2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"
Jul 28 17:44:22.319209 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:22.319179    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spz9m_must-gather-pr486_93ce979f-ca54-4c37-9cb5-b9bc13213159/gather/0.log"
Jul 28 17:44:25.951766 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:25.951733    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bztwt_bea06bea-ce5d-42a6-ad14-3025bee0c0a7/global-pull-secret-syncer/0.log"
Jul 28 17:44:26.195008 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:26.194983    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pzwwh_99abb92a-6ff6-4307-af4f-7f8b0fa3674b/konnectivity-agent/0.log"
Jul 28 17:44:26.254428 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:26.254337    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-17.ec2.internal_a1a690d26e46a9b8ac8c0a59f590243d/haproxy/0.log"
Jul 28 17:44:27.762298 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:27.762267    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-spz9m/must-gather-pr486"]
Jul 28 17:44:27.762781 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:27.762424    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-spz9m/must-gather-pr486" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="copy" containerID="cri-o://bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037" gracePeriod=2
Jul 28 17:44:27.768999 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:27.768973    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-spz9m/must-gather-pr486"]
Jul 28 17:44:27.770175 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:27.770142    2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"93ce979f-ca54-4c37-9cb5-b9bc13213159\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-spz9m\"/\"must-gather-pr486\": pods \"must-gather-pr486\" not found" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:27.995553 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:27.995530    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spz9m_must-gather-pr486_93ce979f-ca54-4c37-9cb5-b9bc13213159/copy/0.log"
Jul 28 17:44:27.998092 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:27.998069    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:28.000137 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:28.000113    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pr486\" is forbidden: User \"system:node:ip-10-0-138-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-spz9m\": no relationship found between node 'ip-10-0-138-17.ec2.internal' and this object" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:28.052236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.052172    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output\") pod \"93ce979f-ca54-4c37-9cb5-b9bc13213159\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") "
Jul 28 17:44:28.052236 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.052216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzxsh\" (UniqueName: \"kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh\") pod \"93ce979f-ca54-4c37-9cb5-b9bc13213159\" (UID: \"93ce979f-ca54-4c37-9cb5-b9bc13213159\") "
Jul 28 17:44:28.053731 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.053710    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "93ce979f-ca54-4c37-9cb5-b9bc13213159" (UID: "93ce979f-ca54-4c37-9cb5-b9bc13213159"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:28.054308 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.054290    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh" (OuterVolumeSpecName: "kube-api-access-lzxsh") pod "93ce979f-ca54-4c37-9cb5-b9bc13213159" (UID: "93ce979f-ca54-4c37-9cb5-b9bc13213159"). InnerVolumeSpecName "kube-api-access-lzxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:28.152822 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.152791    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93ce979f-ca54-4c37-9cb5-b9bc13213159-must-gather-output\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:44:28.152822 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.152816    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxsh\" (UniqueName: \"kubernetes.io/projected/93ce979f-ca54-4c37-9cb5-b9bc13213159-kube-api-access-lzxsh\") on node \"ip-10-0-138-17.ec2.internal\" DevicePath \"\""
Jul 28 17:44:28.848512 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.848486    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spz9m_must-gather-pr486_93ce979f-ca54-4c37-9cb5-b9bc13213159/copy/0.log"
Jul 28 17:44:28.848931 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.848806    2559 generic.go:356] "Generic (PLEG): container finished" podID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerID="bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037" exitCode=143
Jul 28 17:44:28.848931 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.848852    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:28.848931 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.848909    2559 scope.go:117] "RemoveContainer" containerID="bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037"
Jul 28 17:44:28.852105 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:28.852080    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pr486\" is forbidden: User \"system:node:ip-10-0-138-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-spz9m\": no relationship found between node 'ip-10-0-138-17.ec2.internal' and this object" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:28.857549 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.857533    2559 scope.go:117] "RemoveContainer" containerID="2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"
Jul 28 17:44:28.862591 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:28.862560    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pr486\" is forbidden: User \"system:node:ip-10-0-138-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-spz9m\": no relationship found between node 'ip-10-0-138-17.ec2.internal' and this object" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" pod="openshift-must-gather-spz9m/must-gather-pr486"
Jul 28 17:44:28.868390 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.868375    2559 scope.go:117] "RemoveContainer" containerID="bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037"
Jul 28 17:44:28.868640 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:28.868623    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037\": container with ID starting with bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037 not found: ID does not exist" containerID="bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037"
Jul 28 17:44:28.868688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.868650    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037"} err="failed to get container status \"bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037\": rpc error: code = NotFound desc = could not find container \"bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037\": container with ID starting with bf40a9d47e70d5ec3ba93325cff337ff3d18affe4edc822007baa503313a6037 not found: ID does not exist"
Jul 28 17:44:28.868688 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.868666    2559 scope.go:117] "RemoveContainer" containerID="2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"
Jul 28 17:44:28.868923 ip-10-0-138-17 kubenswrapper[2559]: E0728 17:44:28.868903    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64\": container with ID starting with 2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64 not found: ID does not exist" containerID="2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"
Jul 28 17:44:28.868970 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:28.868929    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64"} err="failed to get container status \"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64\": rpc error: code = NotFound desc = could not find container \"2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64\": container with ID starting with 2ef42d7743d6a490a2cc938e063f037f42ac27ea822ed0054eecf18eb7e9dd64 not found: ID does not exist"
Jul 28 17:44:29.672462 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:29.672433    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" path="/var/lib/kubelet/pods/93ce979f-ca54-4c37-9cb5-b9bc13213159/volumes"
Jul 28 17:44:30.128967 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.128933    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-njt9d_d136840f-e10d-4846-9f16-ffbc2ae86c70/cluster-monitoring-operator/0.log"
Jul 28 17:44:30.265218 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.265190    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65975f975f-n4kv6_7cffa9e1-77da-4d2d-b09b-4f6c82d6fc6f/metrics-server/0.log"
Jul 28 17:44:30.299848 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.299821    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-hwn2z_6a3b3e13-869e-4920-b052-f50f8a23f11b/monitoring-plugin/0.log"
Jul 28 17:44:30.542934 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.542854    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbk6s_f79b973c-4627-4f33-bde7-7d7f29a4be69/node-exporter/0.log"
Jul 28 17:44:30.575009 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.574957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbk6s_f79b973c-4627-4f33-bde7-7d7f29a4be69/kube-rbac-proxy/0.log"
Jul 28 17:44:30.597932 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.597910    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbk6s_f79b973c-4627-4f33-bde7-7d7f29a4be69/init-textfile/0.log"
Jul 28 17:44:30.639663 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.639642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m8zfc_c33669e0-c5b9-492c-8f7b-d4bbb74a9e00/kube-rbac-proxy-main/0.log"
Jul 28 17:44:30.671634 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.671611    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m8zfc_c33669e0-c5b9-492c-8f7b-d4bbb74a9e00/kube-rbac-proxy-self/0.log"
Jul 28 17:44:30.700502 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:30.700482    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m8zfc_c33669e0-c5b9-492c-8f7b-d4bbb74a9e00/openshift-state-metrics/0.log"
Jul 28 17:44:31.133584 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.133556    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/thanos-query/0.log"
Jul 28 17:44:31.159617 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.159593    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/kube-rbac-proxy-web/0.log"
Jul 28 17:44:31.188987 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.188961    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/kube-rbac-proxy/0.log"
Jul 28 17:44:31.228835 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.228810    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/prom-label-proxy/0.log"
Jul 28 17:44:31.259460 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.259442    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/kube-rbac-proxy-rules/0.log"
Jul 28 17:44:31.293071 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:31.293053    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-999494bd7-hgj8z_a375a6fc-eaeb-4832-a51a-0277bc208f39/kube-rbac-proxy-metrics/0.log"
Jul 28 17:44:32.576286 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:32.576258    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-8wdww_024d38b7-c1c9-4e99-847d-5d65c15530d7/networking-console-plugin/0.log"
Jul 28 17:44:33.105386 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:33.105359    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/1.log"
Jul 28 17:44:33.110244 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:33.110221    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-8mmmk_5cd5f7ef-ed83-49d5-86c2-7748c28fde5f/console-operator/2.log"
Jul 28 17:44:33.616155 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:33.616128    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-mj5xh_78974b3b-f44f-4b4b-9957-c35bf777fea2/download-server/0.log"
Jul 28 17:44:34.144299 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.144267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-bt9qs_ac10475d-cc6b-4df9-894e-2d750adb3c81/volume-data-source-validator/0.log"
Jul 28 17:44:34.292220 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292190    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"]
Jul 28 17:44:34.292521 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292509    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="copy"
Jul 28 17:44:34.292566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292522    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="copy"
Jul 28 17:44:34.292566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292539    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="gather"
Jul 28 17:44:34.292566 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292544    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="gather"
Jul 28 17:44:34.292664 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292596    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="copy"
Jul 28 17:44:34.292664 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.292607    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ce979f-ca54-4c37-9cb5-b9bc13213159" containerName="gather"
Jul 28 17:44:34.297653 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.297635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.300000 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.299981    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spfd9\"/\"openshift-service-ca.crt\""
Jul 28 17:44:34.300110 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.300070    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spfd9\"/\"kube-root-ca.crt\""
Jul 28 17:44:34.300910 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.300892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-spfd9\"/\"default-dockercfg-992qv\""
Jul 28 17:44:34.303906 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.303882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"]
Jul 28 17:44:34.402351 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.402266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dff\" (UniqueName: \"kubernetes.io/projected/28fb5e83-f14a-4f1f-8986-671bb0054b66-kube-api-access-j7dff\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.402351 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.402329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-sys\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.402539 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.402363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-proc\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.402539 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.402379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-podres\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.402539 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.402431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-lib-modules\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.502891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-sys\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.502891 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-proc\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-podres\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-lib-modules\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dff\" (UniqueName: \"kubernetes.io/projected/28fb5e83-f14a-4f1f-8986-671bb0054b66-kube-api-access-j7dff\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-proc\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.502979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-sys\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.503064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-podres\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.503109 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.503102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28fb5e83-f14a-4f1f-8986-671bb0054b66-lib-modules\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.511113 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.511091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dff\" (UniqueName: \"kubernetes.io/projected/28fb5e83-f14a-4f1f-8986-671bb0054b66-kube-api-access-j7dff\") pod \"perf-node-gather-daemonset-wzthn\" (UID: \"28fb5e83-f14a-4f1f-8986-671bb0054b66\") " pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.607265 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.607240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.741417 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.741391    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"]
Jul 28 17:44:34.743528 ip-10-0-138-17 kubenswrapper[2559]: W0728 17:44:34.743503    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28fb5e83_f14a_4f1f_8986_671bb0054b66.slice/crio-dad3951575958efd36d61a974dcbde18198e70e289d0b12e6a80a800f5fa89da WatchSource:0}: Error finding container dad3951575958efd36d61a974dcbde18198e70e289d0b12e6a80a800f5fa89da: Status 404 returned error can't find the container with id dad3951575958efd36d61a974dcbde18198e70e289d0b12e6a80a800f5fa89da
Jul 28 17:44:34.876744 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.876710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn" event={"ID":"28fb5e83-f14a-4f1f-8986-671bb0054b66","Type":"ContainerStarted","Data":"8e9898dc9bdbb157653c1c38f74c480ba872f9d3f0ba5e048e20f9ed173397dc"}
Jul 28 17:44:34.876908 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.876769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn" event={"ID":"28fb5e83-f14a-4f1f-8986-671bb0054b66","Type":"ContainerStarted","Data":"dad3951575958efd36d61a974dcbde18198e70e289d0b12e6a80a800f5fa89da"}
Jul 28 17:44:34.876908 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.876812    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:34.893746 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:34.893704    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn" podStartSLOduration=0.893688851 podStartE2EDuration="893.688851ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:34.892100322 +0000 UTC m=+2113.774507434" watchObservedRunningTime="2026-07-28 17:44:34.893688851 +0000 UTC m=+2113.776095969"
Jul 28 17:44:35.230703 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:35.230677    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tstvj_ccb53b3e-7d01-4584-b233-2bb144f857be/dns/0.log"
Jul 28 17:44:35.255745 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:35.255720    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tstvj_ccb53b3e-7d01-4584-b233-2bb144f857be/kube-rbac-proxy/0.log"
Jul 28 17:44:35.338851 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:35.338821    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vvb2g_15a784d6-4b9f-48f7-b7af-95afd4bcfce4/dns-node-resolver/0.log"
Jul 28 17:44:35.893354 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:35.893323    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfbzv_66cd0b5b-1bba-4a4b-9a37-79d8e241db87/node-ca/0.log"
Jul 28 17:44:36.649074 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:36.649044    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66cf49fbb7-drwbq_d142323f-846b-471d-82b6-5dfbed8e5128/router/0.log"
Jul 28 17:44:37.033841 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.033766    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n2zb8_ddadacb7-1b96-4c8b-8e13-9f2043680423/serve-healthcheck-canary/0.log"
Jul 28 17:44:37.447618 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.447592    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-sr6tx_17e5be93-25fe-40b8-a784-ba4e359a1f50/insights-operator/0.log"
Jul 28 17:44:37.448152 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.448135    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-sr6tx_17e5be93-25fe-40b8-a784-ba4e359a1f50/insights-operator/1.log"
Jul 28 17:44:37.473435 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.473408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gg8k_42dc4d4c-4a18-4939-99cf-85b7e1b368c3/kube-rbac-proxy/0.log"
Jul 28 17:44:37.497757 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.497732    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gg8k_42dc4d4c-4a18-4939-99cf-85b7e1b368c3/exporter/0.log"
Jul 28 17:44:37.523710 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:37.523689    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gg8k_42dc4d4c-4a18-4939-99cf-85b7e1b368c3/extractor/0.log"
Jul 28 17:44:39.990565 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:39.990540    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-7p44g_0cc56039-5d09-4043-a236-609346f1190c/manager/0.log"
Jul 28 17:44:40.014505 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:40.014482    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dmz9n_e5166ab4-8a8e-49d9-a712-030c67d2324f/server/0.log"
Jul 28 17:44:40.129577 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:40.129553    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-6c5l9_bf4cf9fd-09a2-4c72-8463-4287f91edd29/manager/0.log"
Jul 28 17:44:40.891070 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:40.891045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-spfd9/perf-node-gather-daemonset-wzthn"
Jul 28 17:44:45.005861 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:45.005829    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k_6c594f79-fcb8-4590-bf2f-b4991d614eb8/kube-storage-version-migrator-operator/1.log"
Jul 28 17:44:45.006730 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:45.006707    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tsf8k_6c594f79-fcb8-4590-bf2f-b4991d614eb8/kube-storage-version-migrator-operator/0.log"
Jul 28 17:44:46.417560 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.417527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/kube-multus-additional-cni-plugins/0.log"
Jul 28 17:44:46.442404 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.442384    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/egress-router-binary-copy/0.log"
Jul 28 17:44:46.469431 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.469408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/cni-plugins/0.log"
Jul 28 17:44:46.495717 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.495691    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/bond-cni-plugin/0.log"
Jul 28 17:44:46.522662 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.522641    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/routeoverride-cni/0.log"
Jul 28 17:44:46.547646 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.547626    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/whereabouts-cni-bincopy/0.log"
Jul 28 17:44:46.575001 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.574978    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w8gdg_dab1330b-69d5-4f1c-9266-1ae217f6b1c7/whereabouts-cni/0.log"
Jul 28 17:44:46.642690 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.642658    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fhz2t_5207cf1f-96b1-42b8-8bd1-623cfc669465/kube-multus/0.log"
Jul 28 17:44:46.775463 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.775392    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zk8h7_482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c/network-metrics-daemon/0.log"
Jul 28 17:44:46.798019 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:46.797998    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zk8h7_482a185b-1b91-4c17-bb2c-ad9bf0cc1d6c/kube-rbac-proxy/0.log"
Jul 28 17:44:47.862155 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.862127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/ovn-controller/0.log"
Jul 28 17:44:47.883129 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.883107    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/ovn-acl-logging/0.log"
Jul 28 17:44:47.909823 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.909801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/kube-rbac-proxy-node/0.log"
Jul 28 17:44:47.935857 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.935832    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 17:44:47.958091 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.958072    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/northd/0.log"
Jul 28 17:44:47.982120 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:47.982052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/nbdb/0.log"
Jul 28 17:44:48.006589 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:48.006567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/sbdb/0.log"
Jul 28 17:44:48.053704 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:48.053663    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pmvhk_da4dbe31-ef2d-4a5f-a645-29079110aa50/ovnkube-controller/0.log"
Jul 28 17:44:49.351772 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:49.351736    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-q5m5c_ea489c3b-940c-4ca9-8b3b-111602e74186/check-endpoints/0.log"
Jul 28 17:44:49.405887 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:49.405851    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jtqn7_e663d17a-7350-4a83-9111-8d313abeb23f/network-check-target-container/0.log"
Jul 28 17:44:50.424252 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:50.424223    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-k2qcw_f5af3a59-9f62-4254-b97e-ba7486e1cf54/iptables-alerter/0.log"
Jul 28 17:44:51.187245 ip-10-0-138-17 kubenswrapper[2559]: I0728 17:44:51.187221    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7n85v_26e75bc2-777e-4c20-80fe-5ee167820ece/tuned/0.log"