Jul 27 21:04:31.221536 ip-10-0-131-253 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:04:31.587460 ip-10-0-131-253 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.587460 ip-10-0-131-253 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:04:31.587460 ip-10-0-131-253 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.587460 ip-10-0-131-253 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:04:31.587460 ip-10-0-131-253 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:31.589859 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.589719    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:04:31.594040 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594025    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.594040 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594040    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594044    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594049    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594052    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594055    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594058    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594060    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594063    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594065    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594069    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594073    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594076    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594079    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594082    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594085    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594087    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594090    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.594104 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594092    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594139    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594177    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594182    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594187    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594192    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594198    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594341    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594368    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594374    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594379    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594385    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594396    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594401    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594405    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594410    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594414    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594419    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594423    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.594697 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594432    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594436    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594441    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594444    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594449    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594453    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594458    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594462    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594466    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594470    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594474    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594478    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594486    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594490    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594499    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594503    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594507    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594511    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594515    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594520    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.595373 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594525    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594531    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594535    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594539    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594549    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594553    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594557    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594561    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594565    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594569    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594573    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594577    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594581    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594585    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594590    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594594    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594598    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594607    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594612    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.595851 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594616    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594620    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594624    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594630    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594641    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594648    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594654    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594659    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594663    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594672    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594677    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594682    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594686    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594691    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594695    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594700    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594706    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594711    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594715    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.596325 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594720    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594724    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.594729    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596183    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596195    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596198    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596201    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596205    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596208    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596211    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596213    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596216    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596219    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596222    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596224    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596227    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596230    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596232    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596235    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.596807 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596239    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596241    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596244    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596246    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596248    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596251    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596276    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596278    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596281    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596284    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596286    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596289    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596291    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596294    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596297    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596300    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596303    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596306    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596309    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596311    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.597294 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596314    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596317    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596319    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596322    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596325    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596329    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596333    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596337    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596340    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596343    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596345    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596347    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596350    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596352    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596355    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596357    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596360    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596362    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596365    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596368    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.597777 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596370    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596373    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596375    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596378    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596380    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596383    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596385    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596388    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596390    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596393    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596396    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596399    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596401    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596404    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596406    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596409    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596411    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596414    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596416    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596419    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.598249 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596422    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596425    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596427    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596429    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596432    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596434    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596436    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596439    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596441    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596445    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596448    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596451    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596454    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596456    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596460    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596465    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596468    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596471    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596473    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.598750 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596476    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596478    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596481    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596483    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596910    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596916    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596920    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596923    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596927    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596930    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596933    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596935    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596938    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596940    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596943    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596945    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596948    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596951    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596953    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596956    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.599220 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596958    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596961    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596963    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596965    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596968    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596970    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596972    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596976    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596979    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596981    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596984    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596986    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596989    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596991    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596994    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596996    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.596999    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597001    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597004    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597007    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.599711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597009    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597011    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597014    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597016    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597018    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597021    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597024    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597026    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597029    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597031    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597033    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597036    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597038    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597040    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597043    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597045    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597048    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597050    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597053    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597056    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.600183 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597059    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597061    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597064    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597066    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597069    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597071    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597074    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597078    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597081    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597084    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597086    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597088    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597092    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597095    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597098    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597101    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597103    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597106    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597108    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597111    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.600708 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597113    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597115    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597118    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597123    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597126    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597129    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597133    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597135    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597139    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597142    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597145    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597147    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597150    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597153    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597155    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597158    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597160    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597163    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597165    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.601176 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597168    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597171    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597173    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597176    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597277    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597296    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597306    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597311    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597315    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597318    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597322    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597326    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597329    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597332    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597335    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597339    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597342    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597345    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597348    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597351    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597354    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597356    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597360    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597363    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:04:31.601690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597366    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597371    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597374    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597378    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597381    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597384    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597387    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597390    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597393    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597396    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597399    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597401    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597405    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597408    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597411    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597414    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597417    2580 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597421    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597425    2580 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597428    2580 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597431    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597434    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597437    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597440    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:04:31.602248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597443    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597446    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597449    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597452    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597455    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597458    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597460    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597463    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597466    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597470    2580 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597474    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597476    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597479    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597482    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597485    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597488    2580 flags.go:64] FLAG: --help="false"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597491    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597494    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597497    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597500    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597504    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597507    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597510    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:04:31.602826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597513    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597516    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597519    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597522    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597525    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597528    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597531    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597534    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597537    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597540    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597543    2580 flags.go:64] FLAG: --lock-file=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597545    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597548    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597551    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597556    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597559    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597562    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597565    2580 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597569    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597573    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597575    2580 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597578    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597582    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597585    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597599    2580 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:04:31.603433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597604    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597607    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597610    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597613    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597615    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597618    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597621    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597628    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597631    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597634    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597637    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597640    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597645    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597648    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597651    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597653    2580 flags.go:64] FLAG: --port="10250"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597656    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597659    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a66bb9d66ec7aa70"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597662    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597665    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597668    2580 flags.go:64] FLAG: --register-node="true"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597671    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597674    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597677    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:04:31.604026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597679    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597682    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597687    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597689    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597693    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597695    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597698    2580 flags.go:64] FLAG: --runonce="false"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597701    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597705    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597708    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597711    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597713    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597716    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597719    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597722    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597725    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597728    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597731    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597734    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597750    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597754    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597758    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597768    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597771    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597774    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:04:31.604643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597778    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597781    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597784    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597787    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597789    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597792    2580 flags.go:64] FLAG: --v="2"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597800    2580 flags.go:64] FLAG: --version="false"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597804    2580 flags.go:64] FLAG: --vmodule=""
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597808    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.597812    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597924    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597927    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597930    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597933    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597935    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597939    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597941    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597944    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597946    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597950    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597952    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597954    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.605286 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597957    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597960    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597962    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597965    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597968    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597970    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597973    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597975    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597978    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597980    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597983    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597985    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597988    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597990    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597993    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597996    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.597998    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598001    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598003    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598007    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.605936 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598009    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598012    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598015    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598017    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598019    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598023    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598025    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598029    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598033    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598036    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598039    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598042    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598044    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598046    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598049    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598052    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598056    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598059    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598062    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.606502 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598065    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598067    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598070    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598072    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598075    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598078    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598080    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598082    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598085    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598088    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598090    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598093    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598096    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598099    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598101    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598104    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598106    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598109    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598112    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.606957 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598115    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598117    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598119    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598122    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598124    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598128    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598131    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598134    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598137    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598139    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598142    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598144    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598148    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598150    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598153    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598155    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598158    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598160    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598163    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.607433 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598165    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598167    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598170    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598172    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598174    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598177    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598182    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598185    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598187    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.598190    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.598195    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.604917    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.604933    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.604951    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.604957    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:04:31.607898 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605008    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605013    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605017    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605020    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605023    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605026    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605028    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605031    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605033    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605036    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605038    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605041    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605043    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605046    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605048    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605051    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605053    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605056    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.608278 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605061    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605063    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605066    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605068    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605071    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605073    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605076    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605078    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605081    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605083    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605087    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605089    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605094    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605098    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605101    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605103    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605106    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605109    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605111    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.608780 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605114    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605116    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605120    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605123    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605126    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605129    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605131    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605134    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605136    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605139    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605142    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605145    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605148    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605150    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605152    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605154    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605157    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605159    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605162    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.609239 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605164    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605167    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605169    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605172    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605175    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605178    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605181    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605184    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605186    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605188    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605191    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605193    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605196    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605198    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605201    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605203    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605205    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605209    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605212    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605215    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.609703 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605217    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605219    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605222    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605224    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605227    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605229    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605232    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605234    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605237    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605239    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605241    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605244    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605247    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605249    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605252    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605272    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605275    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605277    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605280    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605283    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.610182 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605285    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605288    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.605293    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605410    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605415    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605418    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605421    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605424    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605427    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605429    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605431    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605434    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605436    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605439    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605441    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:31.610666 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605444    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605446    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605449    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605451    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605454    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605456    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605459    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605461    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605479    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605482    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605486    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605489    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605492    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605495    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605498    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605501    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605504    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605507    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605509    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605512    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:31.611021 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605514    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605517    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605519    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605522    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605524    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605526    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605529    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605531    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605534    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605536    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605538    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605541    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605543    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605546    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605548    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605551    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605553    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605555    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605558    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605560    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:31.611534 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605562    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605565    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605568    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605570    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605573    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605576    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605578    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605581    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605583    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605587    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605589    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605592    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605595    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605599    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605601    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605604    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605607    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605609    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605612    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:31.612009 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605614    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605616    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605619    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605623    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605626    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605629    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605632    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605634    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605637    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605641    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605644    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605646    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605649    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605652    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605655    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605659    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605661    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605664    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605666    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:31.612465 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605669    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605671    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605674    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605676    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605679    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605682    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605684    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605686    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:31.605688    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.605693    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.605793    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.607929    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.609194    2580 server.go:1019] "Starting client certificate rotation"
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.609282    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:31.612920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.609317    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:31.613268 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.612392    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:31.613512 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.613497    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:31.624020 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.624000    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:04:31.626106 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.626091    2580 log.go:25] "Validated CRI v1 image API"
Jul 27 21:04:31.627544 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.627524    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:04:31.629817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.629790    2580 fs.go:135] Filesystem UUIDs: map[0fbb9982-1609-4e16-800a-8bbcf326016e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 842d4b5a-e791-4274-b962-3c740a55bcd1:/dev/nvme0n1p3]
Jul 27 21:04:31.629887 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.629816    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:04:31.635432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.635322    2580 manager.go:216] Machine: {Timestamp:2026-07-27 21:04:31.634067954 +0000 UTC m=+0.298693203 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099711 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b01f0c56cb2a7b2268ac336195f6d SystemUUID:ec2b01f0-c56c-b2a7-b226-8ac336195f6d BootID:fd937151-8d3b-4773-aaae-56263785dba5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:e7:2a:a7:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:e7:2a:a7:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:97:11:77:25:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:04:31.635432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.635424    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:04:31.635556 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.635499    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:04:31.637291 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.637275    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:31.638298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.638272    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:04:31.638465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.638305    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:04:31.639017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.639006    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:04:31.639103 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.639022    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:04:31.639103 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.639032    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:31.639690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.639677    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:31.640873 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.640862    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:04:31.643995 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.643981    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:31.644122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644111    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:04:31.644178 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644150    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:04:31.644178 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644171    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:04:31.644293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644191    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:04:31.644293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644206    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:04:31.644293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644219    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:04:31.644606 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644592    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:31.644667 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.644615    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:31.646617 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.646596    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:04:31.648462 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.648449    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:04:31.648512 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.648479    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:04:31.650221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650208    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:04:31.650277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650226    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:04:31.650277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650232    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:04:31.650277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650238    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:04:31.650277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650246    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:04:31.650277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650252    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650281    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650286    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650293    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650298    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650311    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:04:31.650417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.650316    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:04:31.651033 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.651025    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:04:31.651033 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.651033    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:04:31.654852 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.654833    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:04:31.655777 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.655755    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:04:31.655826 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.655759    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:04:31.655826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.655809    2580 server.go:1268] "Started kubelet"
Jul 27 21:04:31.655929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.655905    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:04:31.656220 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.656152    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:04:31.656464 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.656443    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:04:31.656868 ip-10-0-131-253 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:04:31.661451 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.661427    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fspfz"
Jul 27 21:04:31.661540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.661463    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:04:31.661589 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.661552    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:04:31.662039 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.662018    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:04:31.664673 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.663824    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-253.ec2.internal.18c64174abf25837  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-253.ec2.internal,UID:ip-10-0-131-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-253.ec2.internal,},FirstTimestamp:2026-07-27 21:04:31.655778359 +0000 UTC m=+0.320403588,LastTimestamp:2026-07-27 21:04:31.655778359 +0000 UTC m=+0.320403588,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-253.ec2.internal,}"
Jul 27 21:04:31.666795 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.666782    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:04:31.666862 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.666796    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:31.666921 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.666860    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:31.667518 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667503    2580 factory.go:55] Registering systemd factory
Jul 27 21:04:31.667650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667639    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:04:31.667737 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667508    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:04:31.667834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667509    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:04:31.667834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667826    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:04:31.667967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667903    2580 factory.go:153] Registering CRI-O factory
Jul 27 21:04:31.667967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667916    2580 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:04:31.667967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667934    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:04:31.667967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667943    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:04:31.667967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667964    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:04:31.668140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.667990    2580 factory.go:103] Registering Raw factory
Jul 27 21:04:31.668140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.668006    2580 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:04:31.668406 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.668385    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fspfz"
Jul 27 21:04:31.668510 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.668440    2580 manager.go:349] Starting recovery of all containers
Jul 27 21:04:31.679326 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.679181    2580 manager.go:354] Recovery completed
Jul 27 21:04:31.679845 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.679830    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:31.683301 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.683285    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-253.ec2.internal\" not found" node="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.683574 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.683561    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.685383 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685368    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.685461 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685395    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.685461 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685405    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.685875 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685861    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:04:31.685875 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685875    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:04:31.685994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685894    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:31.685994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.685922    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.686701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.686688    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.686755 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.686712    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.686755 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.686721    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.687514 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.687501    2580 policy_none.go:49] "None policy: Start"
Jul 27 21:04:31.687580 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.687520    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:04:31.687580 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.687533    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:04:31.688452 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.688440    2580 policy_none.go:47] "Start"
Jul 27 21:04:31.722770 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.722755    2580 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:04:31.722865 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.722786    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:04:31.722865 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.722796    2580 server.go:87] "Starting device plugin registration server"
Jul 27 21:04:31.723148 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.723035    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:04:31.723196 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.723154    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:04:31.723292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.723276    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:04:31.723377 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.723366    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:04:31.723450 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.723377    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:04:31.723891 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.723868    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:04:31.723939 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.723927    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:31.823483 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.823456    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.824800 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.824361    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.824800 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.824391    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.824800 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.824406    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.824800 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.824430    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.836814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.836798    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.836869 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.836822    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-253.ec2.internal\": node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:31.853234 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.853186    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:31.870157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.870133    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:04:31.871524 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.871507    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:04:31.871612 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.871541    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:04:31.871612 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.871561    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:04:31.871612 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.871590    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:04:31.874418 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.874401    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:31.954168 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:31.954144    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:31.972608 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.972589    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"]
Jul 27 21:04:31.972660 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.972649    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.974174 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.974158    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.974241 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.974190    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.974241 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.974203    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.975744 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.975732    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.975874 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.975860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.975934 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.975890    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.976429 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976408    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.976536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976442    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.976536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976457    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.976536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976410    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.976536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976507    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.976536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.976515    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:31.977859 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.977846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:31.977904 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.977869    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:31.978505 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.978488    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:31.978602 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.978518    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:31.978602 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:31.978532    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:32.002923 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.002904    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-253.ec2.internal\" not found" node="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.007105 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.007089    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-253.ec2.internal\" not found" node="ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.054742 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.054719    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.069604 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.069587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.069685 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.069611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1c1da79ad55c5026a8b619f1f45eec5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"f1c1da79ad55c5026a8b619f1f45eec5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.069685 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.069629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.155606 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.155554    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.169903 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.169970 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.169970 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1c1da79ad55c5026a8b619f1f45eec5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"f1c1da79ad55c5026a8b619f1f45eec5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.169970 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1c1da79ad55c5026a8b619f1f45eec5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"f1c1da79ad55c5026a8b619f1f45eec5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.170069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.170069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.169991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0bcbc251df60d8bac11a67c074ab50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"0a0bcbc251df60d8bac11a67c074ab50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.255873 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.255843    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.305311 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.305290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.311108 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.311092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:32.356764 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.356740    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.457282 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.457215    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.495639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.495612    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:32.558248 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.558228    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.608755 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.608726    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:04:32.609152 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.608872    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:04:32.609152 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.608899    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:04:32.609152 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.608937    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:04:32.658768 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.658748    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.672799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.672765    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:59:32 +0000 UTC" deadline="2028-04-29 21:13:03.776370028 +0000 UTC"
Jul 27 21:04:32.672799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.672796    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15408h8m31.103577257s"
Jul 27 21:04:32.759565 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.759519    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.860611 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.860567    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:32.875206 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:32.875176    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0bcbc251df60d8bac11a67c074ab50.slice/crio-6cf82a1f563c2ef115262689dea90b5277365b1f659f86de54a4c46ec29b1ac1 WatchSource:0}: Error finding container 6cf82a1f563c2ef115262689dea90b5277365b1f659f86de54a4c46ec29b1ac1: Status 404 returned error can't find the container with id 6cf82a1f563c2ef115262689dea90b5277365b1f659f86de54a4c46ec29b1ac1
Jul 27 21:04:32.875719 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:32.875696    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c1da79ad55c5026a8b619f1f45eec5.slice/crio-22d91f9c2b7f087959ba33793aeb608c50fef0a980144f172fef64ec499013dc WatchSource:0}: Error finding container 22d91f9c2b7f087959ba33793aeb608c50fef0a980144f172fef64ec499013dc: Status 404 returned error can't find the container with id 22d91f9c2b7f087959ba33793aeb608c50fef0a980144f172fef64ec499013dc
Jul 27 21:04:32.881175 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:32.881160    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:04:32.961797 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:32.961764    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:33.067399 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.067373    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found"
Jul 27 21:04:33.140222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.140190    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.167067 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.167036    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:33.180025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.180007    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:33.181060 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.181047    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal"
Jul 27 21:04:33.193014 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.192997    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:33.458530 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.458453    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.645695 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.645660    2580 apiserver.go:52] "Watching apiserver"
Jul 27 21:04:33.655742 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.655708    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:04:33.656214 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.656182    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d","openshift-cluster-node-tuning-operator/tuned-9wzmp","openshift-dns/node-resolver-856bp","openshift-image-registry/node-ca-hf45z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","openshift-multus/multus-additional-cni-plugins-wzkd7","openshift-network-diagnostics/network-check-target-rcjn8","kube-system/global-pull-secret-syncer-zhdtx","kube-system/konnectivity-agent-59lqd","openshift-multus/multus-4dppf","openshift-multus/network-metrics-daemon-phzrj","openshift-network-operator/iptables-alerter-zk6x4","openshift-ovn-kubernetes/ovnkube-node-rk54c","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"]
Jul 27 21:04:33.658294 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.658252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:33.658422 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.658360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:33.659881 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.659857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.659982 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.659935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:33.661322 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.661297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.663017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.663000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.665954 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.665933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.666065 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.666013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:33.666399 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.666373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.666512 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.666380    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-frqgl\""
Jul 27 21:04:33.666997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.666975    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.667494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.667073    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:04:33.667494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.667206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.667494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.667269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.667494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.667283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rc7bb\""
Jul 27 21:04:33.667494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.667435    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:33.669500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.669099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.670783 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.670762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.673237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.672852    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:04:33.673237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.673027    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.673711 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.673429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:04:33.673711 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.673472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:04:33.673816 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.673714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lfffh\""
Jul 27 21:04:33.673816 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.673718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:04:33.675327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.674003    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.675327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.674208    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7mcgk\""
Jul 27 21:04:33.675327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.674450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:04:33.675327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.674704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.675704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.675687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.676596 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.676571    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.677508 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.677487    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.677695 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.677509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.677695 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.677516    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2gtb2\""
Jul 27 21:04:33.677695 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.677531    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:04:33.678272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.678252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:04:33.678435 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.678322    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:04:33.678543 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.678514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:04:33.678979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.678957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-kubelet-config\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.679061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.678993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-cnibin\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.679061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1796f663-d2a4-4f37-9442-3ea4ead2efc4-tmp-dir\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjp8\" (UniqueName: \"kubernetes.io/projected/1796f663-d2a4-4f37-9442-3ea4ead2efc4-kube-api-access-gsjp8\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gk4\" (UniqueName: \"kubernetes.io/projected/8e468562-5b4a-418b-8ae4-360be809fde2-kube-api-access-f8gk4\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-os-release\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679259 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5wh\" (UniqueName: \"kubernetes.io/projected/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-kube-api-access-ql5wh\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e468562-5b4a-418b-8ae4-360be809fde2-serviceca\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclzw\" (UniqueName: \"kubernetes.io/projected/63836abf-191c-49be-ab1c-98f96564118d-kube-api-access-zclzw\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1796f663-d2a4-4f37-9442-3ea4ead2efc4-hosts-file\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e468562-5b4a-418b-8ae4-360be809fde2-host\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-dbus\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.679650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.679609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.680759 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.680609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.681202 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.681181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.681377 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.680611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.681528 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.681509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:04:33.682616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.682323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b72jn\""
Jul 27 21:04:33.682616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.682581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xp7r2\""
Jul 27 21:04:33.683202 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.683183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.683436 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.683415    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.683524 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.683481    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpvq8\""
Jul 27 21:04:33.683650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.683632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:04:33.684716 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.684688    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8nn7m\""
Jul 27 21:04:33.684809 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.684694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:33.685468 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.685046    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:33.685911 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.685805    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:33.686347 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.686326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:04:33.705130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.705096    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-llhx2"
Jul 27 21:04:33.706506 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.706484    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:33.716526 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.716499    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-llhx2"
Jul 27 21:04:33.769610 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.769554    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:04:33.779824 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.779785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.779824 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.779821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.780056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.779849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.780056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.779880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cni-binary-copy\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.780056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.779940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-config\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.780056 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.780004    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28c9r\" (UniqueName: \"kubernetes.io/projected/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kube-api-access-28c9r\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.780124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.280096159 +0000 UTC m=+2.944721390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-sys\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-var-lib-kubelet\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-tuned\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-multus\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-host-slash\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-slash\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjsb\" (UniqueName: \"kubernetes.io/projected/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-kube-api-access-jfjsb\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclzw\" (UniqueName: \"kubernetes.io/projected/63836abf-191c-49be-ab1c-98f96564118d-kube-api-access-zclzw\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-kubelet\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-netd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-registration-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-host\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-tmp\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.780644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-ovn\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1796f663-d2a4-4f37-9442-3ea4ead2efc4-hosts-file\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e468562-5b4a-418b-8ae4-360be809fde2-host\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-node-log\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1796f663-d2a4-4f37-9442-3ea4ead2efc4-hosts-file\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e468562-5b4a-418b-8ae4-360be809fde2-host\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-sys-fs\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-cnibin\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-cnibin\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-etc-kubernetes\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-systemd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjp8\" (UniqueName: \"kubernetes.io/projected/1796f663-d2a4-4f37-9442-3ea4ead2efc4-kube-api-access-gsjp8\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-os-release\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.780998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cnibin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5wh\" (UniqueName: \"kubernetes.io/projected/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-kube-api-access-ql5wh\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e468562-5b4a-418b-8ae4-360be809fde2-serviceca\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-os-release\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-netns\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovn-node-metrics-cert\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-run\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-modprobe-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-kubernetes\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-os-release\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-conf-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbdr\" (UniqueName: \"kubernetes.io/projected/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-kube-api-access-knbdr\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc57acf-9d20-4b82-b522-8a44e559a4d4-konnectivity-ca\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.781706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-device-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysconfig\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gk4\" (UniqueName: \"kubernetes.io/projected/8e468562-5b4a-418b-8ae4-360be809fde2-kube-api-access-f8gk4\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-kubelet\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-etc-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-log-socket\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e468562-5b4a-418b-8ae4-360be809fde2-serviceca\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-script-lib\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-k8s-cni-cncf-io\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.781982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-dbus\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmhx\" (UniqueName: \"kubernetes.io/projected/147e8328-dcf4-4551-bdb1-d8b84bd08858-kube-api-access-qxmhx\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-socket-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.782234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-conf\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-kubelet-config\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-hostroot\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-daemon-config\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc57acf-9d20-4b82-b522-8a44e559a4d4-agent-certs\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-var-lib-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-systemd\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-lib-modules\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1796f663-d2a4-4f37-9442-3ea4ead2efc4-tmp-dir\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-dbus\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-socket-dir-parent\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63836abf-191c-49be-ab1c-98f96564118d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.782291    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-bin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsnj\" (UniqueName: \"kubernetes.io/projected/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-kube-api-access-qpsnj\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2eec02d3-3654-484f-9f70-60fbac97a123-kubelet-config\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:33.782853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-bin\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-env-overrides\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.782549    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.282532321 +0000 UTC m=+2.947157551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-system-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-multus-certs\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1796f663-d2a4-4f37-9442-3ea4ead2efc4-tmp-dir\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-iptables-alerter-script\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-systemd-units\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63836abf-191c-49be-ab1c-98f96564118d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.783426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.782835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-netns\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.797553 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.797524    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:33.797553 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.797556    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:33.797797 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.797570    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:33.797797 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:33.797635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.297619585 +0000 UTC m=+2.962244802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:33.799728 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.799683    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:04:33.804233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.804205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gk4\" (UniqueName: \"kubernetes.io/projected/8e468562-5b4a-418b-8ae4-360be809fde2-kube-api-access-f8gk4\") pod \"node-ca-hf45z\" (UID: \"8e468562-5b4a-418b-8ae4-360be809fde2\") " pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.804362 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.804275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclzw\" (UniqueName: \"kubernetes.io/projected/63836abf-191c-49be-ab1c-98f96564118d-kube-api-access-zclzw\") pod \"multus-additional-cni-plugins-wzkd7\" (UID: \"63836abf-191c-49be-ab1c-98f96564118d\") " pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:33.804475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.804452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5wh\" (UniqueName: \"kubernetes.io/projected/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-kube-api-access-ql5wh\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:33.804879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.804857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjp8\" (UniqueName: \"kubernetes.io/projected/1796f663-d2a4-4f37-9442-3ea4ead2efc4-kube-api-access-gsjp8\") pod \"node-resolver-856bp\" (UID: \"1796f663-d2a4-4f37-9442-3ea4ead2efc4\") " pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.877594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.877529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"0a0bcbc251df60d8bac11a67c074ab50","Type":"ContainerStarted","Data":"6cf82a1f563c2ef115262689dea90b5277365b1f659f86de54a4c46ec29b1ac1"}
Jul 27 21:04:33.878862 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.878819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"f1c1da79ad55c5026a8b619f1f45eec5","Type":"ContainerStarted","Data":"22d91f9c2b7f087959ba33793aeb608c50fef0a980144f172fef64ec499013dc"}
Jul 27 21:04:33.883135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.883260 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-run\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.883260 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-modprobe-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-kubernetes\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-os-release\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-conf-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-run\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.883972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbdr\" (UniqueName: \"kubernetes.io/projected/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-kube-api-access-knbdr\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-conf-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc57acf-9d20-4b82-b522-8a44e559a4d4-konnectivity-ca\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.884147 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-os-release\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-modprobe-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-kubernetes\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-device-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysconfig\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-kubelet\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-device-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-etc-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysconfig\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-log-socket\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-script-lib\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc57acf-9d20-4b82-b522-8a44e559a4d4-konnectivity-ca\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-k8s-cni-cncf-io\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-log-socket\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-k8s-cni-cncf-io\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-etc-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmhx\" (UniqueName: \"kubernetes.io/projected/147e8328-dcf4-4551-bdb1-d8b84bd08858-kube-api-access-qxmhx\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.885790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.884997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-socket-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-kubelet\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-socket-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-conf\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-conf\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-hostroot\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-daemon-config\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc57acf-9d20-4b82-b522-8a44e559a4d4-agent-certs\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-var-lib-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-systemd\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-lib-modules\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-socket-dir-parent\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-bin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsnj\" (UniqueName: \"kubernetes.io/projected/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-kube-api-access-qpsnj\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-script-lib\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-bin\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-env-overrides\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-var-lib-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-system-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-systemd\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-multus-certs\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-iptables-alerter-script\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-systemd-units\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-lib-modules\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-netns\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-socket-dir-parent\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-daemon-config\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-hostroot\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.885969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-multus-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-multus-certs\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.888850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cni-binary-copy\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-bin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-config\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-systemd-units\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28c9r\" (UniqueName: \"kubernetes.io/projected/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kube-api-access-28c9r\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-sys\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-bin\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-var-lib-kubelet\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-run-netns\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-tuned\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-sys\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-multus\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-host-slash\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-slash\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-slash\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-cni-multus\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-host-slash\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-system-cni-dir\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.889659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjsb\" (UniqueName: \"kubernetes.io/projected/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-kube-api-access-jfjsb\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-kubelet\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cni-binary-copy\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-netd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-var-lib-kubelet\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-iptables-alerter-script\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-registration-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-host-cni-netd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-var-lib-kubelet\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovnkube-config\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.886896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-registration-dir\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-host\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-tmp\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-ovn\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-node-log\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-host\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-sys-fs\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.890466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-sys-fs\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-etc-kubernetes\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-systemd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cnibin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-netns\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/147e8328-dcf4-4551-bdb1-d8b84bd08858-env-overrides\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovn-node-metrics-cert\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-systemd\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-etc-kubernetes\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-host-run-netns\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-cnibin\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-ovn\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.887954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-node-log\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.888006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/147e8328-dcf4-4551-bdb1-d8b84bd08858-run-openvswitch\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.888210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-sysctl-d\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.889055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-etc-tuned\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.889531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc57acf-9d20-4b82-b522-8a44e559a4d4-agent-certs\") pod \"konnectivity-agent-59lqd\" (UID: \"2dc57acf-9d20-4b82-b522-8a44e559a4d4\") " pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:33.891293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.890463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/147e8328-dcf4-4551-bdb1-d8b84bd08858-ovn-node-metrics-cert\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.892116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.892041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-tmp\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.895651 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.895595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbdr\" (UniqueName: \"kubernetes.io/projected/1d1049b4-69ae-445c-bae0-4ca9de4a6cbb-kube-api-access-knbdr\") pod \"multus-4dppf\" (UID: \"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb\") " pod="openshift-multus/multus-4dppf"
Jul 27 21:04:33.896633 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.896607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjsb\" (UniqueName: \"kubernetes.io/projected/dac5a78f-0a74-45bb-88f4-5a83d35b14ff-kube-api-access-jfjsb\") pod \"tuned-9wzmp\" (UID: \"dac5a78f-0a74-45bb-88f4-5a83d35b14ff\") " pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:33.896857 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.896833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsnj\" (UniqueName: \"kubernetes.io/projected/b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c-kube-api-access-qpsnj\") pod \"iptables-alerter-zk6x4\" (UID: \"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c\") " pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:33.897219 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.897197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmhx\" (UniqueName: \"kubernetes.io/projected/147e8328-dcf4-4551-bdb1-d8b84bd08858-kube-api-access-qxmhx\") pod \"ovnkube-node-rk54c\" (UID: \"147e8328-dcf4-4551-bdb1-d8b84bd08858\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:33.898328 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.898306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28c9r\" (UniqueName: \"kubernetes.io/projected/b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac-kube-api-access-28c9r\") pod \"aws-ebs-csi-driver-node-m5f7d\" (UID: \"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:33.977535 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.977440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-856bp"
Jul 27 21:04:33.986770 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.986747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf45z"
Jul 27 21:04:33.997498 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:33.997479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzkd7"
Jul 27 21:04:34.003450 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.003430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:34.012378 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.012356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:34.020251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.020231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9wzmp"
Jul 27 21:04:34.027319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.027301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4dppf"
Jul 27 21:04:34.035048 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.035025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d"
Jul 27 21:04:34.042636 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.042610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk6x4"
Jul 27 21:04:34.290995 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.290899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:34.290995 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.290952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:34.291212 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.291066    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:34.291212 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.291068    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.291212 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.291128    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.291110809 +0000 UTC m=+3.955736027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:34.291212 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.291144    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.291137728 +0000 UTC m=+3.955762946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:34.391319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.391261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:34.391513 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.391411    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:34.391513 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.391435    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:34.391513 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.391450    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:34.391664 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.391516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:35.391496569 +0000 UTC m=+4.056121787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:34.719320 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.719253    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:34 +0000 UTC" deadline="2028-01-18 14:29:40.304205322 +0000 UTC"
Jul 27 21:04:34.719320 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.719311    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12953h25m5.58489916s"
Jul 27 21:04:34.872449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.872424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:34.872556 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.872449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:34.872556 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:34.872450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:34.872556 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.872544    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:34.872662 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.872614    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:34.872744 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:34.872725    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:34.892059 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.892032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ee99da_ae7a_4ca9_8704_fb0a5c3173ac.slice/crio-8fab15670dbc9ecebd40f62f950030eab4b82c4af2aa72c4a1dcb818e2f0d1dd WatchSource:0}: Error finding container 8fab15670dbc9ecebd40f62f950030eab4b82c4af2aa72c4a1dcb818e2f0d1dd: Status 404 returned error can't find the container with id 8fab15670dbc9ecebd40f62f950030eab4b82c4af2aa72c4a1dcb818e2f0d1dd
Jul 27 21:04:34.893232 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.893206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1796f663_d2a4_4f37_9442_3ea4ead2efc4.slice/crio-50e378b8a2280e9546d1c97f78fb9aa48abde6371677a1e786c5e36c515f62c5 WatchSource:0}: Error finding container 50e378b8a2280e9546d1c97f78fb9aa48abde6371677a1e786c5e36c515f62c5: Status 404 returned error can't find the container with id 50e378b8a2280e9546d1c97f78fb9aa48abde6371677a1e786c5e36c515f62c5
Jul 27 21:04:34.897944 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.897908    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e468562_5b4a_418b_8ae4_360be809fde2.slice/crio-b7ac979556b68b65af0e9e9d8dcc50eb5450cbd5e2e8bce7df078115dde7b4a4 WatchSource:0}: Error finding container b7ac979556b68b65af0e9e9d8dcc50eb5450cbd5e2e8bce7df078115dde7b4a4: Status 404 returned error can't find the container with id b7ac979556b68b65af0e9e9d8dcc50eb5450cbd5e2e8bce7df078115dde7b4a4
Jul 27 21:04:34.898945 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.898924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f92be2_dda7_4dd3_a43c_0f4ca41ecb2c.slice/crio-1157656c04c9971c0c94d43cacd6f1a4036586177b76e282b09db54b3b5ef868 WatchSource:0}: Error finding container 1157656c04c9971c0c94d43cacd6f1a4036586177b76e282b09db54b3b5ef868: Status 404 returned error can't find the container with id 1157656c04c9971c0c94d43cacd6f1a4036586177b76e282b09db54b3b5ef868
Jul 27 21:04:34.899453 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.899261    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147e8328_dcf4_4551_bdb1_d8b84bd08858.slice/crio-a94cb2a19e51315db69be212a5b823bc1ec73e0b798b58624d16bd81d2982b59 WatchSource:0}: Error finding container a94cb2a19e51315db69be212a5b823bc1ec73e0b798b58624d16bd81d2982b59: Status 404 returned error can't find the container with id a94cb2a19e51315db69be212a5b823bc1ec73e0b798b58624d16bd81d2982b59
Jul 27 21:04:34.901122 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.900994    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63836abf_191c_49be_ab1c_98f96564118d.slice/crio-3562f9335b48586521e9948089d8068a64ad15c52d1e2be0447bbbada83016e8 WatchSource:0}: Error finding container 3562f9335b48586521e9948089d8068a64ad15c52d1e2be0447bbbada83016e8: Status 404 returned error can't find the container with id 3562f9335b48586521e9948089d8068a64ad15c52d1e2be0447bbbada83016e8
Jul 27 21:04:34.902343 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.902204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc57acf_9d20_4b82_b522_8a44e559a4d4.slice/crio-773b08a4980111db69f7687cdb663c1a25e68e95b869540edf3954e34085635a WatchSource:0}: Error finding container 773b08a4980111db69f7687cdb663c1a25e68e95b869540edf3954e34085635a: Status 404 returned error can't find the container with id 773b08a4980111db69f7687cdb663c1a25e68e95b869540edf3954e34085635a
Jul 27 21:04:34.903173 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.903101    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1049b4_69ae_445c_bae0_4ca9de4a6cbb.slice/crio-23407e243ad1338e5aa518830dc4a88ba45312405e608de90cbea45b995bfc85 WatchSource:0}: Error finding container 23407e243ad1338e5aa518830dc4a88ba45312405e608de90cbea45b995bfc85: Status 404 returned error can't find the container with id 23407e243ad1338e5aa518830dc4a88ba45312405e608de90cbea45b995bfc85
Jul 27 21:04:34.905664 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:04:34.905625    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac5a78f_0a74_45bb_88f4_5a83d35b14ff.slice/crio-f8156a591e28451408890463e7e02edb1f72925ce43c16c7011fdbf5b0ef35c4 WatchSource:0}: Error finding container f8156a591e28451408890463e7e02edb1f72925ce43c16c7011fdbf5b0ef35c4: Status 404 returned error can't find the container with id f8156a591e28451408890463e7e02edb1f72925ce43c16c7011fdbf5b0ef35c4
Jul 27 21:04:35.178370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.178347    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:35.298751 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.298649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:35.298751 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.298702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:35.298944 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.298832    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:35.298944 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.298915    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:35.298944 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.298911    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.298889028 +0000 UTC m=+5.963514248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:35.299105 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.298966    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.298952286 +0000 UTC m=+5.963577503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:35.399470 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.399429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:35.399667 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.399612    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:35.399667 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.399633    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:35.399667 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.399646    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:35.399836 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:35.399710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.399690951 +0000 UTC m=+6.064316170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:35.720054 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.720010    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:34 +0000 UTC" deadline="2028-02-02 11:58:53.575564035 +0000 UTC"
Jul 27 21:04:35.720054 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.720052    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13310h54m17.855516903s"
Jul 27 21:04:35.900622 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.900458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d" event={"ID":"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac","Type":"ContainerStarted","Data":"8fab15670dbc9ecebd40f62f950030eab4b82c4af2aa72c4a1dcb818e2f0d1dd"}
Jul 27 21:04:35.910390 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.909669    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a0bcbc251df60d8bac11a67c074ab50" containerID="5b0a3370b333bd256669b1e22baf70c1b21aa027692dfbcdbca025972af8471d" exitCode=0
Jul 27 21:04:35.910390 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.909754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"0a0bcbc251df60d8bac11a67c074ab50","Type":"ContainerDied","Data":"5b0a3370b333bd256669b1e22baf70c1b21aa027692dfbcdbca025972af8471d"}
Jul 27 21:04:35.922304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.922240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4dppf" event={"ID":"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb","Type":"ContainerStarted","Data":"23407e243ad1338e5aa518830dc4a88ba45312405e608de90cbea45b995bfc85"}
Jul 27 21:04:35.938621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.938588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerStarted","Data":"3562f9335b48586521e9948089d8068a64ad15c52d1e2be0447bbbada83016e8"}
Jul 27 21:04:35.956626 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.956593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk6x4" event={"ID":"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c","Type":"ContainerStarted","Data":"1157656c04c9971c0c94d43cacd6f1a4036586177b76e282b09db54b3b5ef868"}
Jul 27 21:04:35.967202 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.967177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf45z" event={"ID":"8e468562-5b4a-418b-8ae4-360be809fde2","Type":"ContainerStarted","Data":"b7ac979556b68b65af0e9e9d8dcc50eb5450cbd5e2e8bce7df078115dde7b4a4"}
Jul 27 21:04:35.981383 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:35.981352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"f1c1da79ad55c5026a8b619f1f45eec5","Type":"ContainerStarted","Data":"038d21a0634014f7022909c95a24786705be28a9ab643340e7a2c9cf1dc39b71"}
Jul 27 21:04:36.011493 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.011463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9wzmp" event={"ID":"dac5a78f-0a74-45bb-88f4-5a83d35b14ff","Type":"ContainerStarted","Data":"f8156a591e28451408890463e7e02edb1f72925ce43c16c7011fdbf5b0ef35c4"}
Jul 27 21:04:36.026837 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.026350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59lqd" event={"ID":"2dc57acf-9d20-4b82-b522-8a44e559a4d4","Type":"ContainerStarted","Data":"773b08a4980111db69f7687cdb663c1a25e68e95b869540edf3954e34085635a"}
Jul 27 21:04:36.031389 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.031280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"a94cb2a19e51315db69be212a5b823bc1ec73e0b798b58624d16bd81d2982b59"}
Jul 27 21:04:36.041011 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.040981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-856bp" event={"ID":"1796f663-d2a4-4f37-9442-3ea4ead2efc4","Type":"ContainerStarted","Data":"50e378b8a2280e9546d1c97f78fb9aa48abde6371677a1e786c5e36c515f62c5"}
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.872495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:36.872648    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.872729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:36.872802    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:36.872865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:36.873315 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:36.872953    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:37.081558 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:37.081405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"0a0bcbc251df60d8bac11a67c074ab50","Type":"ContainerStarted","Data":"c3def617f5c9cc615c7a25ffdf504c1e9cfd28a3caa973a31d2895e1ed8059e3"}
Jul 27 21:04:37.102891 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:37.101794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" podStartSLOduration=4.101771227 podStartE2EDuration="4.101771227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:36.009609962 +0000 UTC m=+4.674235233" watchObservedRunningTime="2026-07-27 21:04:37.101771227 +0000 UTC m=+5.766396470"
Jul 27 21:04:37.316590 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:37.316544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:37.316804 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:37.316620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:37.316804 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.316791    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:37.316922 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.316864    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.316841853 +0000 UTC m=+9.981467095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:37.317300 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.317279    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:37.317406 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.317357    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.317322436 +0000 UTC m=+9.981947661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:37.417382 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:37.417331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:37.417560 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.417509    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:37.417560 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.417533    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:37.417560 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.417547    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:37.417726 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:37.417608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:41.417588299 +0000 UTC m=+10.082213521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:38.872466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:38.872610    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:38.873024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:38.873125    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:38.873193    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:38.873330 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:38.873282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:40.872670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:40.872434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:40.872670 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:40.872551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:40.872670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:40.872583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:40.872670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:40.872597    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:40.872670 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:40.872649    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:40.873229 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:40.872973    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:41.350272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:41.350212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:41.350467 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:41.350291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:41.350467 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.350377    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:41.350467 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.350444    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.350425468 +0000 UTC m=+18.015050699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:41.350467 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.350459    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:41.350690 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.350520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.350500565 +0000 UTC m=+18.015125799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:41.451273 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:41.451203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:41.451532 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.451509    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:41.451634 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.451539    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:41.451634 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.451552    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:41.451634 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:41.451612    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:49.451592747 +0000 UTC m=+18.116217981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:42.872544 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:42.872505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:42.873010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:42.872505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:42.873010 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:42.872635    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:42.873010 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:42.872726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:42.873010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:42.872514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:42.873010 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:42.872816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:44.872197 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:44.872168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:44.872644 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:44.872313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:44.872704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:44.872688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:44.872808 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:44.872781    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:44.872919 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:44.872847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:44.872977 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:44.872927    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:46.872217 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:46.872182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:46.872629 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:46.872315    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:46.872679 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:46.872628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:46.872679 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:46.872652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:46.872750 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:46.872723    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:46.872815 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:46.872787    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:48.872683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:48.872649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:48.872683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:48.872661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:48.872683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:48.872685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:48.873379 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:48.872761    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:48.873379 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:48.872844    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:48.873379 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:48.872942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:49.414460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:49.414426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:49.414641 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:49.414526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:49.414641 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.414536    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:49.414641 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.414606    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.414584886 +0000 UTC m=+34.079210104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:49.414641 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.414611    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:49.414833 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.414661    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.414644318 +0000 UTC m=+34.079269541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:49.515788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:49.515761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:49.515981 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.515937    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:49.515981 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.515960    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:49.515981 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.515972    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:49.516133 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:49.516033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:05:05.51601183 +0000 UTC m=+34.180637066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:50.872662 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:50.872624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:50.873092 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:50.872678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:50.873092 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:50.872639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:50.873092 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:50.872743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:50.873092 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:50.872799    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:50.873092 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:50.872882    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:52.872072 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:52.872039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:52.872072 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:52.872066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:52.872508 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:52.872072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:52.872508 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:52.872157    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:52.872508 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:52.872217    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:52.872508 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:52.872306    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:54.872388 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:54.872002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:54.873188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:54.872015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:54.873188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:54.872071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:54.873188 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:54.872542    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:54.873188 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:54.872661    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:54.873188 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:54.872748    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:55.120375 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.120339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4dppf" event={"ID":"1d1049b4-69ae-445c-bae0-4ca9de4a6cbb","Type":"ContainerStarted","Data":"9219d918f88938922f3dc66815761510d6dcd8fceae0725d28aaf2b10045e948"}
Jul 27 21:04:55.122012 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.121957    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="aea8606741a6540ab3873bb9828a2f5fb60372fd3510ef0dea39a642a8971a98" exitCode=0
Jul 27 21:04:55.122119 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.122035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"aea8606741a6540ab3873bb9828a2f5fb60372fd3510ef0dea39a642a8971a98"}
Jul 27 21:04:55.123835 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.123802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf45z" event={"ID":"8e468562-5b4a-418b-8ae4-360be809fde2","Type":"ContainerStarted","Data":"ef5c1187f3c8a4484592f32f8c3ea823395bf28d28750d39d198cd7953474940"}
Jul 27 21:04:55.135601 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.135576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9wzmp" event={"ID":"dac5a78f-0a74-45bb-88f4-5a83d35b14ff","Type":"ContainerStarted","Data":"c1ad45c35813e4a4017b108ef17dfd618b872851686f0e978022904fb333ec7b"}
Jul 27 21:04:55.136906 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.136883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59lqd" event={"ID":"2dc57acf-9d20-4b82-b522-8a44e559a4d4","Type":"ContainerStarted","Data":"2b9bb34fe6fbbe436ae1a60c6682298cbc9ce18d74b6ae5fd7d9cee077a6b653"}
Jul 27 21:04:55.137752 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.137721    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4dppf" podStartSLOduration=3.891445547 podStartE2EDuration="23.13771018s" totalImagesPullingTime="19.246264633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.137502507 +0000 UTC m=+23.802127748" watchObservedRunningTime="2026-07-27 21:04:55.13771018 +0000 UTC m=+23.802335418"
Jul 27 21:04:55.138291 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.138245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" podStartSLOduration=22.138233322 podStartE2EDuration="22.138233322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:37.102835444 +0000 UTC m=+5.767460703" watchObservedRunningTime="2026-07-27 21:04:55.138233322 +0000 UTC m=+23.802858562"
Jul 27 21:04:55.139837 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"89d627bdd8fcd0b545618c6df800069f8b3626a60b92738473ab7bf2bf5d9084"}
Jul 27 21:04:55.139930 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"bbaaf5566093a78c7f7ecbe78d389095ee6390f696ab05cf94bf77fe32c39a7e"}
Jul 27 21:04:55.139930 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"80fbb4f92863a4ecf62b9d2facacf4a4717519ce25885c0f80f084536bf1646e"}
Jul 27 21:04:55.140025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"a89df0a09befea09cad67497c2e7f41a15324ab58c9d0be0b881ffc65a5a58f2"}
Jul 27 21:04:55.140025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"c37b3310a25ac5a814ff11e25eba0e076252f4895f7b539a34f98b4f62dc587b"}
Jul 27 21:04:55.140025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.139998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"92c41dbc487b78ef5889405a40ff948081ad38db32eca03241204c6050301201"}
Jul 27 21:04:55.141073 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.141054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-856bp" event={"ID":"1796f663-d2a4-4f37-9442-3ea4ead2efc4","Type":"ContainerStarted","Data":"447010645c43f8fcc863505e6650c9428ff16afdec8e9d7412437543d5474a75"}
Jul 27 21:04:55.142380 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.142356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d" event={"ID":"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac","Type":"ContainerStarted","Data":"aa2e9279df21fb552b39ed92c8b4b86baf606c472d430008fa86f0375b0793fe"}
Jul 27 21:04:55.175051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.175018    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-59lqd" podStartSLOduration=3.97825463 podStartE2EDuration="23.175007361s" totalImagesPullingTime="19.196752731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.174883846 +0000 UTC m=+23.839509083" watchObservedRunningTime="2026-07-27 21:04:55.175007361 +0000 UTC m=+23.839632599"
Jul 27 21:04:55.192448 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.192418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hf45z" podStartSLOduration=3.991071362 podStartE2EDuration="23.192407737s" totalImagesPullingTime="19.201336375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.192154795 +0000 UTC m=+23.856780032" watchObservedRunningTime="2026-07-27 21:04:55.192407737 +0000 UTC m=+23.857032975"
Jul 27 21:04:55.213975 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.213945    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9wzmp" podStartSLOduration=4.022262572 podStartE2EDuration="23.213935693s" totalImagesPullingTime="19.191673121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.213440197 +0000 UTC m=+23.878065434" watchObservedRunningTime="2026-07-27 21:04:55.213935693 +0000 UTC m=+23.878560931"
Jul 27 21:04:55.231723 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.231686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-856bp" podStartSLOduration=4.026144746 podStartE2EDuration="23.231675875s" totalImagesPullingTime="19.205531129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:55.23124772 +0000 UTC m=+23.895872960" watchObservedRunningTime="2026-07-27 21:04:55.231675875 +0000 UTC m=+23.896301114"
Jul 27 21:04:55.527008 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.526981    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:04:55.741398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.741214    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:04:55.527005451Z","UUID":"6c59936a-47d9-4687-a449-3afd933d2775","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:04:55.742769 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.742745    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:04:55.742880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:55.742780    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:04:56.147585 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:56.147547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d" event={"ID":"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac","Type":"ContainerStarted","Data":"c56538e9e66c554c52c3cd4fb5b55eb0d6e9d3b7c360f2b656b25a4f9eea689d"}
Jul 27 21:04:56.150359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:56.150329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk6x4" event={"ID":"b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c","Type":"ContainerStarted","Data":"150768f44fd2134aeb1198e5a8d18998fa8b61a1a85cd8a157964acf8155d03d"}
Jul 27 21:04:56.872486 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:56.872462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:56.872594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:56.872498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:56.872594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:56.872499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:56.872691 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:56.872600    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:56.872691 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:56.872675    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:56.872785 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:56.872737    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:57.156714 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.156679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"d09cb6907638515fca050c6fab2211cc209b02448626299d7812b36f9cfb750a"}
Jul 27 21:04:57.159749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.159723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d" event={"ID":"b7ee99da-ae7a-4ca9-8704-fb0a5c3173ac","Type":"ContainerStarted","Data":"25b705bae101405517bd404b0176d2d856559f56b755ff53597c199adb93c13c"}
Jul 27 21:04:57.190246 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.190199    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zk6x4" podStartSLOduration=5.984806828 podStartE2EDuration="25.190179925s" totalImagesPullingTime="19.205373097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:56.167383907 +0000 UTC m=+24.832009145" watchObservedRunningTime="2026-07-27 21:04:57.190179925 +0000 UTC m=+25.854805164"
Jul 27 21:04:57.190458 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.190435    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5f7d" podStartSLOduration=3.744113143 podStartE2EDuration="25.190429441s" totalImagesPullingTime="21.446316298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:57.190099237 +0000 UTC m=+25.854724502" watchObservedRunningTime="2026-07-27 21:04:57.190429441 +0000 UTC m=+25.855054676"
Jul 27 21:04:57.517378 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.517306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:57.517819 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:57.517802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:58.161798 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:58.161731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:58.162174 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:58.162127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-59lqd"
Jul 27 21:04:58.872759 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:58.872724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:04:58.872916 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:58.872763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:04:58.872916 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:58.872763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:04:58.872916 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:58.872838    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:04:58.873134 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:58.872922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:04:58.873134 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:04:58.873002    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:04:59.170122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:59.168269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" event={"ID":"147e8328-dcf4-4551-bdb1-d8b84bd08858","Type":"ContainerStarted","Data":"4145be31cf114c0b3ac3905aeb7b74106f1fac6a516cff3d3bacb56ddb4ff258"}
Jul 27 21:04:59.170122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:59.168777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:59.185571 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:59.185349    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:04:59.263300 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:04:59.263225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c" podStartSLOduration=8.001143335 podStartE2EDuration="27.263203192s" totalImagesPullingTime="19.262059857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.219534868 +0000 UTC m=+27.884160123" watchObservedRunningTime="2026-07-27 21:04:59.263203192 +0000 UTC m=+27.927828430"
Jul 27 21:05:00.172135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.172093    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="bf060e89de880165869c81294f589449eb05abb03519d71ba684f5867253e411" exitCode=0
Jul 27 21:05:00.172798 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.172135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"bf060e89de880165869c81294f589449eb05abb03519d71ba684f5867253e411"}
Jul 27 21:05:00.172798 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.172736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:05:00.172905 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.172815    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:05:00.188348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.188331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:05:00.872554 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.872382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:00.872766 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:00.872658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:05:00.873098 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.873078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:00.873211 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:00.873193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:05:00.873298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:00.873284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:00.873393 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:00.873374    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:05:01.454957 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.454864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhdtx"]
Jul 27 21:05:01.455333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.455038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:01.455333 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:01.455151    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:05:01.458539 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.458514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rcjn8"]
Jul 27 21:05:01.458688 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.458565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-phzrj"]
Jul 27 21:05:01.458688 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.458646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:01.458788 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:01.458744    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:05:01.458845 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:01.458809    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:01.458887 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:01.458873    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:05:02.181735 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:02.181694    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="f499c6f9579fc47e7712dcfdff48c643ccd2bda09cd04309e1c393f5b6db05c9" exitCode=0
Jul 27 21:05:02.181917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:02.181785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"f499c6f9579fc47e7712dcfdff48c643ccd2bda09cd04309e1c393f5b6db05c9"}
Jul 27 21:05:02.872100 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:02.872070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:02.872100 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:02.872099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:02.872586 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:02.872198    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:05:02.872586 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:02.872359    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:05:03.875753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:03.875582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:03.876098 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:03.875838    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:05:04.188376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:04.188291    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="aa436c6464704e56e7f38aecbf026cae4c4716bf239309c898b0d20cdb4107d2" exitCode=0
Jul 27 21:05:04.188376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:04.188334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"aa436c6464704e56e7f38aecbf026cae4c4716bf239309c898b0d20cdb4107d2"}
Jul 27 21:05:04.872198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:04.872167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:04.872378 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:04.872305    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zhdtx" podUID="2eec02d3-3654-484f-9f70-60fbac97a123"
Jul 27 21:05:04.872378 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:04.872367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:04.872488 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:04.872468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phzrj" podUID="9a84e1cd-17c3-43dc-b4d7-47c93e63686f"
Jul 27 21:05:05.435779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:05.435740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:05.436206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:05.435797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:05.436206 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.435879    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:05.436206 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.435939    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret podName:2eec02d3-3654-484f-9f70-60fbac97a123 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.435920395 +0000 UTC m=+66.100545624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret") pod "global-pull-secret-syncer-zhdtx" (UID: "2eec02d3-3654-484f-9f70-60fbac97a123") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:05.436206 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.435959    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:05.436206 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.436008    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.435991529 +0000 UTC m=+66.100616760 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:05.536454 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:05.536423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:05.536621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.536574    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:05:05.536621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.536591    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:05:05.536621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.536600    2580 projected.go:196] Error preparing data for projected volume kube-api-access-xdjb2 for pod openshift-network-diagnostics/network-check-target-rcjn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:05.536753 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.536642    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2 podName:1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb nodeName:}" failed. No retries permitted until 2026-07-27 21:05:37.53663068 +0000 UTC m=+66.201255896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdjb2" (UniqueName: "kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2") pod "network-check-target-rcjn8" (UID: "1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:05.875806 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:05.875684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:05.875979 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:05.875935    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rcjn8" podUID="1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb"
Jul 27 21:05:06.645726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.645695    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeReady"
Jul 27 21:05:06.646056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.645830    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:05:06.694189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.694155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6skq5"]
Jul 27 21:05:06.696558 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.696538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.699247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.699223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:05:06.699361 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.699300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:05:06.699612 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.699597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5hshq\""
Jul 27 21:05:06.699881 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.699858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:05:06.703214 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.703196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bmpfv"]
Jul 27 21:05:06.705006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.704986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.706576 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.706554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6skq5"]
Jul 27 21:05:06.708316 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.707703    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:05:06.708316 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.707744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:05:06.708316 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.707750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k6b4w\""
Jul 27 21:05:06.719484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.719445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmpfv"]
Jul 27 21:05:06.844221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-cert\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.844221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6hh\" (UniqueName: \"kubernetes.io/projected/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-kube-api-access-xv6hh\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.844465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.844465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae68659-399b-4891-a9f8-bf55fae841a8-tmp-dir\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.844465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae68659-399b-4891-a9f8-bf55fae841a8-config-volume\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.844465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.844397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2hj\" (UniqueName: \"kubernetes.io/projected/9ae68659-399b-4891-a9f8-bf55fae841a8-kube-api-access-7l2hj\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.872376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.872345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:06.872508 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.872345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:06.875135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.875109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r4zfp\""
Jul 27 21:05:06.875276 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.875135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:06.875326 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.875293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:06.944838 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.944803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-cert\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.944838 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.944844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6hh\" (UniqueName: \"kubernetes.io/projected/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-kube-api-access-xv6hh\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.945182 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.945293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae68659-399b-4891-a9f8-bf55fae841a8-tmp-dir\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.945293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae68659-399b-4891-a9f8-bf55fae841a8-config-volume\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.945293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2hj\" (UniqueName: \"kubernetes.io/projected/9ae68659-399b-4891-a9f8-bf55fae841a8-kube-api-access-7l2hj\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.945617 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:06.945597    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:06.945718 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:06.945668    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:07.445648762 +0000 UTC m=+36.110273985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:06.945781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae68659-399b-4891-a9f8-bf55fae841a8-tmp-dir\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.945998 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.945971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae68659-399b-4891-a9f8-bf55fae841a8-config-volume\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.949654 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.949635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-cert\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:06.956409 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.956389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2hj\" (UniqueName: \"kubernetes.io/projected/9ae68659-399b-4891-a9f8-bf55fae841a8-kube-api-access-7l2hj\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:06.956610 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:06.956593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6hh\" (UniqueName: \"kubernetes.io/projected/a6506781-41f2-4f90-ad2b-a0d7f9ccc627-kube-api-access-xv6hh\") pod \"ingress-canary-6skq5\" (UID: \"a6506781-41f2-4f90-ad2b-a0d7f9ccc627\") " pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:07.008475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.008437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6skq5"
Jul 27 21:05:07.450782 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.450748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:07.450983 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:07.450909    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:07.450983 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:07.450975    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:08.450953991 +0000 UTC m=+37.115579211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:07.876541 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.876509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:07.880611 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.880590    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:07.880886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.880868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p5l9j\""
Jul 27 21:05:07.881004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:07.880982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:08.179378 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:08.179224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6skq5"]
Jul 27 21:05:08.182689 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:08.182662    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6506781_41f2_4f90_ad2b_a0d7f9ccc627.slice/crio-da9e9427b9ab23d48f2d4df2b5af41ce5b81482aeed76d063f9801e021003903 WatchSource:0}: Error finding container da9e9427b9ab23d48f2d4df2b5af41ce5b81482aeed76d063f9801e021003903: Status 404 returned error can't find the container with id da9e9427b9ab23d48f2d4df2b5af41ce5b81482aeed76d063f9801e021003903
Jul 27 21:05:08.199555 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:08.199518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6skq5" event={"ID":"a6506781-41f2-4f90-ad2b-a0d7f9ccc627","Type":"ContainerStarted","Data":"da9e9427b9ab23d48f2d4df2b5af41ce5b81482aeed76d063f9801e021003903"}
Jul 27 21:05:08.202415 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:08.202389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerStarted","Data":"083edeef15f0b8e73b8a637c115c7e58a11e52f2e7b657fdb1d8a596dd07b938"}
Jul 27 21:05:08.458686 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:08.458573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:08.458837 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:08.458713    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:08.458837 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:08.458780    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:10.458763763 +0000 UTC m=+39.123388981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:09.207936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:09.207900    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="083edeef15f0b8e73b8a637c115c7e58a11e52f2e7b657fdb1d8a596dd07b938" exitCode=0
Jul 27 21:05:09.208537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:09.207968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"083edeef15f0b8e73b8a637c115c7e58a11e52f2e7b657fdb1d8a596dd07b938"}
Jul 27 21:05:10.213883 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:10.213845    2580 generic.go:356] "Generic (PLEG): container finished" podID="63836abf-191c-49be-ab1c-98f96564118d" containerID="42f2ecab71d83cac90c8c46625f4cd3a233754e72d5a58a0ae721508b8bdfb37" exitCode=0
Jul 27 21:05:10.214324 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:10.213895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerDied","Data":"42f2ecab71d83cac90c8c46625f4cd3a233754e72d5a58a0ae721508b8bdfb37"}
Jul 27 21:05:10.475794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:10.475733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:10.475893 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:10.475870    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:10.475937 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:10.475922    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:14.475907437 +0000 UTC m=+43.140532658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:11.217798 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:11.217606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6skq5" event={"ID":"a6506781-41f2-4f90-ad2b-a0d7f9ccc627","Type":"ContainerStarted","Data":"32636dd35ee7ecdb2bfc698a15cdfdf34ad9402f666f6354ace1a2688ba8819f"}
Jul 27 21:05:11.223536 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:11.220960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" event={"ID":"63836abf-191c-49be-ab1c-98f96564118d","Type":"ContainerStarted","Data":"eb5dd16d792c04305a2d01139257ee0c36780651fbecc25fa29b6e429c0bc6f4"}
Jul 27 21:05:11.238170 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:11.238129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6skq5" podStartSLOduration=2.944238765 podStartE2EDuration="5.238116709s" totalImagesPullingTime="2.293877944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:11.237460175 +0000 UTC m=+39.902085433" watchObservedRunningTime="2026-07-27 21:05:11.238116709 +0000 UTC m=+39.902741947"
Jul 27 21:05:11.263303 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:11.263238    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wzkd7" podStartSLOduration=10.077037228 podStartE2EDuration="39.263219734s" totalImagesPullingTime="29.186182506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:11.260823254 +0000 UTC m=+39.925448491" watchObservedRunningTime="2026-07-27 21:05:11.263219734 +0000 UTC m=+39.927844973"
Jul 27 21:05:14.501730 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:14.501691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:14.502108 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:14.501841    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:14.502108 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:14.501902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:22.50188647 +0000 UTC m=+51.166511688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:20.271005 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.270977    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"]
Jul 27 21:05:20.287310 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.287276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"]
Jul 27 21:05:20.287438 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.287363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"
Jul 27 21:05:20.289823 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.289802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 21:05:20.290861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.290837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:05:20.290979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.290863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hq7xc\""
Jul 27 21:05:20.397119 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.397059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"]
Jul 27 21:05:20.411559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.411417    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-cb9c7cff9-gn874"]
Jul 27 21:05:20.411713 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.411694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"
Jul 27 21:05:20.414271 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.414239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nfcjw\""
Jul 27 21:05:20.430105 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.430086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"]
Jul 27 21:05:20.430215 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.430118    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cb9c7cff9-gn874"]
Jul 27 21:05:20.430295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.430220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pj45f\""
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 21:05:20.433243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433120    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 21:05:20.433585 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.433571    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 21:05:20.439908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.439890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wszk\" (UniqueName: \"kubernetes.io/projected/1ae13130-fc45-41e6-8dc9-65418018048b-kube-api-access-8wszk\") pod \"volume-data-source-validator-549c6d54cf-csc2v\" (UID: \"1ae13130-fc45-41e6-8dc9-65418018048b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"
Jul 27 21:05:20.472020 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.471998    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"]
Jul 27 21:05:20.485304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.485287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.485903 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.485880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"]
Jul 27 21:05:20.488046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.487947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 27 21:05:20.488046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.487965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 27 21:05:20.488046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.487978    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:05:20.488046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.487999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 27 21:05:20.488317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.488074    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xz8dd\""
Jul 27 21:05:20.540836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wszk\" (UniqueName: \"kubernetes.io/projected/1ae13130-fc45-41e6-8dc9-65418018048b-kube-api-access-8wszk\") pod \"volume-data-source-validator-549c6d54cf-csc2v\" (UID: \"1ae13130-fc45-41e6-8dc9-65418018048b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"
Jul 27 21:05:20.540836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.540972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lsv\" (UniqueName: \"kubernetes.io/projected/18926344-1fbe-4113-b076-97a3ee1f0705-kube-api-access-z8lsv\") pod \"network-check-source-7d88cb55d8-sbdd7\" (UID: \"18926344-1fbe-4113-b076-97a3ee1f0705\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"
Jul 27 21:05:20.540972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-default-certificate\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.540972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-stats-auth\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.540972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.540972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.540961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5h7\" (UniqueName: \"kubernetes.io/projected/ff98cefc-9b31-4473-bd21-68d1dc2b191d-kube-api-access-7w5h7\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.558296 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.558271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wszk\" (UniqueName: \"kubernetes.io/projected/1ae13130-fc45-41e6-8dc9-65418018048b-kube-api-access-8wszk\") pod \"volume-data-source-validator-549c6d54cf-csc2v\" (UID: \"1ae13130-fc45-41e6-8dc9-65418018048b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"
Jul 27 21:05:20.575273 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.575230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-wtrtk"]
Jul 27 21:05:20.590559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.590538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ff6b98d77-zvbvf"]
Jul 27 21:05:20.590675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.590658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.593327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.593092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q6srw\""
Jul 27 21:05:20.593437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.593383    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:05:20.593437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.593402    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:05:20.593437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.593430    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 27 21:05:20.593624 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.593605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 27 21:05:20.596066 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.596047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"
Jul 27 21:05:20.598770 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.598750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 27 21:05:20.604037 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.604020    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-wtrtk"]
Jul 27 21:05:20.604121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.604046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff6b98d77-zvbvf"]
Jul 27 21:05:20.604121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.604102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.607522 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.607503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 21:05:20.607608 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.607553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 21:05:20.608356 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.608338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 21:05:20.613282 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.613248    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d6g8r\""
Jul 27 21:05:20.614435 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.614414    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 21:05:20.642059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.642188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.642188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lsv\" (UniqueName: \"kubernetes.io/projected/18926344-1fbe-4113-b076-97a3ee1f0705-kube-api-access-z8lsv\") pod \"network-check-source-7d88cb55d8-sbdd7\" (UID: \"18926344-1fbe-4113-b076-97a3ee1f0705\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"
Jul 27 21:05:20.642188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-default-certificate\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.642188 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.642172    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-stats-auth\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-config\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwl2\" (UniqueName: \"kubernetes.io/projected/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-kube-api-access-wfwl2\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.642346    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:21.142323983 +0000 UTC m=+49.806949207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:20.642423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.642385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5h7\" (UniqueName: \"kubernetes.io/projected/ff98cefc-9b31-4473-bd21-68d1dc2b191d-kube-api-access-7w5h7\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.642702 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.642482    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:21.142470527 +0000 UTC m=+49.807095757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:20.644602 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.644582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-default-certificate\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.644739 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.644722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-stats-auth\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.652039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.652015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5h7\" (UniqueName: \"kubernetes.io/projected/ff98cefc-9b31-4473-bd21-68d1dc2b191d-kube-api-access-7w5h7\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:20.660705 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.660678    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lsv\" (UniqueName: \"kubernetes.io/projected/18926344-1fbe-4113-b076-97a3ee1f0705-kube-api-access-z8lsv\") pod \"network-check-source-7d88cb55d8-sbdd7\" (UID: \"18926344-1fbe-4113-b076-97a3ee1f0705\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"
Jul 27 21:05:20.721139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.721114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"
Jul 27 21:05:20.729619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.729599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v"]
Jul 27 21:05:20.731794 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:20.731771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae13130_fc45_41e6_8dc9_65418018048b.slice/crio-db85148f6bf0f8f401f1f320b92f5377c2dc990960908459ca6ffe0ebef1d116 WatchSource:0}: Error finding container db85148f6bf0f8f401f1f320b92f5377c2dc990960908459ca6ffe0ebef1d116: Status 404 returned error can't find the container with id db85148f6bf0f8f401f1f320b92f5377c2dc990960908459ca6ffe0ebef1d116
Jul 27 21:05:20.743588 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-snapshots\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.743669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-tmp\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.743669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.743736 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.743772 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84spd\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.743803 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwl2\" (UniqueName: \"kubernetes.io/projected/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-kube-api-access-wfwl2\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.743803 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.743886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ca8d50-1f85-489a-a938-779325f199b1-serving-cert\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.743886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/f8ca8d50-1f85-489a-a938-779325f199b1-kube-api-access-76j9d\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.743966 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.743966 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.743947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.744038 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.744077 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-service-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.744077 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-config\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.744171 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.744221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.744293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.744563 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.744542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-config\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.746064 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.746047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-serving-cert\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.752080 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.752056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwl2\" (UniqueName: \"kubernetes.io/projected/cc3dc3a3-3420-49b2-9e28-76219d7f91bb-kube-api-access-wfwl2\") pod \"service-ca-operator-5d6c75f84f-pb4s4\" (UID: \"cc3dc3a3-3420-49b2-9e28-76219d7f91bb\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.794407 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.794353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"
Jul 27 21:05:20.844997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.844962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/f8ca8d50-1f85-489a-a938-779325f199b1-kube-api-access-76j9d\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.845137 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845137 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.845137 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-service-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.845336    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.845351    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-snapshots\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:20.845419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:21.345400075 +0000 UTC m=+50.010025311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-tmp\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84spd\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.845842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ca8d50-1f85-489a-a938-779325f199b1-serving-cert\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.845941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-service-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-snapshots\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8ca8d50-1f85-489a-a938-779325f199b1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.846491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8ca8d50-1f85-489a-a938-779325f199b1-tmp\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.846940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.846675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.850888 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.847933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.850888 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.848699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.852064 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.852024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8ca8d50-1f85-489a-a938-779325f199b1-serving-cert\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.854043 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.854022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7"]
Jul 27 21:05:20.857494 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.857471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.857607 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:20.857507    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18926344_1fbe_4113_b076_97a3ee1f0705.slice/crio-76bb4846e01997a6057d786972b3cab1e28232556504052ab5d9867c47dd7cff WatchSource:0}: Error finding container 76bb4846e01997a6057d786972b3cab1e28232556504052ab5d9867c47dd7cff: Status 404 returned error can't find the container with id 76bb4846e01997a6057d786972b3cab1e28232556504052ab5d9867c47dd7cff
Jul 27 21:05:20.857673 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.857648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j9d\" (UniqueName: \"kubernetes.io/projected/f8ca8d50-1f85-489a-a938-779325f199b1-kube-api-access-76j9d\") pod \"insights-operator-5885fdb759-wtrtk\" (UID: \"f8ca8d50-1f85-489a-a938-779325f199b1\") " pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.857841 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.857818    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84spd\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:20.900572 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.900547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-wtrtk"
Jul 27 21:05:20.946701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:20.946657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4"]
Jul 27 21:05:20.948631 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:20.948602    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3dc3a3_3420_49b2_9e28_76219d7f91bb.slice/crio-0583f527f84a9a82abad632438b8d1eaf674d5edbb768988bacf354775565db0 WatchSource:0}: Error finding container 0583f527f84a9a82abad632438b8d1eaf674d5edbb768988bacf354775565db0: Status 404 returned error can't find the container with id 0583f527f84a9a82abad632438b8d1eaf674d5edbb768988bacf354775565db0
Jul 27 21:05:21.048460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.048389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-wtrtk"]
Jul 27 21:05:21.062924 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:21.062899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ca8d50_1f85_489a_a938_779325f199b1.slice/crio-454354e096e2ca482f5c27162a9845c5aa7b879cb7e3d8c3becd9f2ca0f3f713 WatchSource:0}: Error finding container 454354e096e2ca482f5c27162a9845c5aa7b879cb7e3d8c3becd9f2ca0f3f713: Status 404 returned error can't find the container with id 454354e096e2ca482f5c27162a9845c5aa7b879cb7e3d8c3becd9f2ca0f3f713
Jul 27 21:05:21.147794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.147771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:21.147899 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.147819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:21.147937 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.147916    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:22.147903266 +0000 UTC m=+50.812528483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:21.147937 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.147917    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:21.148008 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.147974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:22.147956717 +0000 UTC m=+50.812581933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:21.247956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.247924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-wtrtk" event={"ID":"f8ca8d50-1f85-489a-a938-779325f199b1","Type":"ContainerStarted","Data":"454354e096e2ca482f5c27162a9845c5aa7b879cb7e3d8c3becd9f2ca0f3f713"}
Jul 27 21:05:21.249307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.249278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7" event={"ID":"18926344-1fbe-4113-b076-97a3ee1f0705","Type":"ContainerStarted","Data":"76bb4846e01997a6057d786972b3cab1e28232556504052ab5d9867c47dd7cff"}
Jul 27 21:05:21.250468 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.250441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v" event={"ID":"1ae13130-fc45-41e6-8dc9-65418018048b","Type":"ContainerStarted","Data":"db85148f6bf0f8f401f1f320b92f5377c2dc990960908459ca6ffe0ebef1d116"}
Jul 27 21:05:21.251637 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.251616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4" event={"ID":"cc3dc3a3-3420-49b2-9e28-76219d7f91bb","Type":"ContainerStarted","Data":"0583f527f84a9a82abad632438b8d1eaf674d5edbb768988bacf354775565db0"}
Jul 27 21:05:21.349186 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:21.349163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:21.349621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.349312    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:21.349621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.349331    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:21.349621 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:21.349395    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:22.349374269 +0000 UTC m=+51.013999488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:22.158961 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:22.158925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:22.159136 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:22.158988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:22.159136 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.159090    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:22.159136 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.159121    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:24.159102616 +0000 UTC m=+52.823727838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:22.159299 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.159145    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:24.15912926 +0000 UTC m=+52.823754477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:22.360880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:22.360829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:22.361412 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.360996    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:22.361412 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.361012    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:22.361412 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.361071    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:24.361058058 +0000 UTC m=+53.025683276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:22.562122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:22.562068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:22.562327 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.562224    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:22.562327 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:22.562309    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls podName:9ae68659-399b-4891-a9f8-bf55fae841a8 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:38.562289964 +0000 UTC m=+67.226915179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls") pod "dns-default-bmpfv" (UID: "9ae68659-399b-4891-a9f8-bf55fae841a8") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:23.259408 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:23.259376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v" event={"ID":"1ae13130-fc45-41e6-8dc9-65418018048b","Type":"ContainerStarted","Data":"968ab362c1eb2e833172045b4967f1e17a290d3c609507e41cd73d71aef96244"}
Jul 27 21:05:23.277403 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:23.277360    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-csc2v" podStartSLOduration=1.597903564 podStartE2EDuration="3.277346027s" totalImagesPullingTime="1.679442463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:23.277208873 +0000 UTC m=+51.941834137" watchObservedRunningTime="2026-07-27 21:05:23.277346027 +0000 UTC m=+51.941971242"
Jul 27 21:05:24.174565 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:24.174518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:24.174998 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:24.174607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:24.174998 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.174623    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:24.174998 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.174712    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:28.174692159 +0000 UTC m=+56.839317378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:24.174998 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.174736    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:28.174725836 +0000 UTC m=+56.839351052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:24.376428 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:24.376394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:24.376582 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.376510    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:24.376582 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.376522    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:24.376582 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:24.376572    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:28.376558218 +0000 UTC m=+57.041183435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:26.268223 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.268182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-wtrtk" event={"ID":"f8ca8d50-1f85-489a-a938-779325f199b1","Type":"ContainerStarted","Data":"7f110cace7ec345f13f2de9bbbdfd1433478b13f5fc971b5c034ee17c3d23505"}
Jul 27 21:05:26.269960 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.269927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7" event={"ID":"18926344-1fbe-4113-b076-97a3ee1f0705","Type":"ContainerStarted","Data":"4682825af71dc19771f917a609bab1b407349f6f7ce43e01775b210faf1981f2"}
Jul 27 21:05:26.271707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.271682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4" event={"ID":"cc3dc3a3-3420-49b2-9e28-76219d7f91bb","Type":"ContainerStarted","Data":"3f2281347ef5d21dc59c41b81d2feeed575b97de8a0e9c1ff0042a62f799da05"}
Jul 27 21:05:26.289086 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.289025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-wtrtk" podStartSLOduration=1.95229162 podStartE2EDuration="6.289006576s" totalImagesPullingTime="4.336714956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:26.28743618 +0000 UTC m=+54.952061420" watchObservedRunningTime="2026-07-27 21:05:26.289006576 +0000 UTC m=+54.953631811"
Jul 27 21:05:26.311056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.311005    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-sbdd7" podStartSLOduration=1.7694640659999998 podStartE2EDuration="6.310991293s" totalImagesPullingTime="4.541527227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:26.310556141 +0000 UTC m=+54.975181380" watchObservedRunningTime="2026-07-27 21:05:26.310991293 +0000 UTC m=+54.975616531"
Jul 27 21:05:26.328151 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.328105    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4" podStartSLOduration=1.871908125 podStartE2EDuration="6.328088057s" totalImagesPullingTime="4.456179932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:26.327055862 +0000 UTC m=+54.991681097" watchObservedRunningTime="2026-07-27 21:05:26.328088057 +0000 UTC m=+54.992713296"
Jul 27 21:05:26.701394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.701363    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxbp"]
Jul 27 21:05:26.704116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.704093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.706659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.706600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:05:26.706659 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.706634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:05:26.706808 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.706759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j726j\""
Jul 27 21:05:26.719632 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.719610    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxbp"]
Jul 27 21:05:26.795002 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.794974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/445ace8f-0db8-449f-916a-0daaa96c9691-crio-socket\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.795002 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.795004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qr2j\" (UniqueName: \"kubernetes.io/projected/445ace8f-0db8-449f-916a-0daaa96c9691-kube-api-access-7qr2j\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.795168 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.795041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/445ace8f-0db8-449f-916a-0daaa96c9691-data-volume\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.795168 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.795142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.795325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.795246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896563 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/445ace8f-0db8-449f-916a-0daaa96c9691-data-volume\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/445ace8f-0db8-449f-916a-0daaa96c9691-crio-socket\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr2j\" (UniqueName: \"kubernetes.io/projected/445ace8f-0db8-449f-916a-0daaa96c9691-kube-api-access-7qr2j\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.896870 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:26.896762    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:26.896870 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:26.896785    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:26.896870 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:26.896839    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:27.396818841 +0000 UTC m=+56.061444077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:26.896870 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:26.896859    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:27.39684909 +0000 UTC m=+56.061474311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:26.897174 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/445ace8f-0db8-449f-916a-0daaa96c9691-crio-socket\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.897174 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.896947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/445ace8f-0db8-449f-916a-0daaa96c9691-data-volume\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:26.904805 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:26.904784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qr2j\" (UniqueName: \"kubernetes.io/projected/445ace8f-0db8-449f-916a-0daaa96c9691-kube-api-access-7qr2j\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:27.400702 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:27.400669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:27.401059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:27.400753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:27.401059 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:27.400781    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:27.401059 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:27.400843    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:28.400826663 +0000 UTC m=+57.065451887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:27.401059 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:27.400853    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:27.401059 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:27.400888    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:28.400877656 +0000 UTC m=+57.065502872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:28.207827 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:28.207798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:28.208014 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:28.207866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:28.208014 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.207928    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:28.208014 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.207993    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:36.207978187 +0000 UTC m=+64.872603403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:28.208014 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.208007    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:36.208001339 +0000 UTC m=+64.872626555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:28.409337 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:28.409309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:28.409345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:28.409418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409430    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409484    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:30.4094675 +0000 UTC m=+59.074092719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409515    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409521    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409538    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409571    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:30.409559034 +0000 UTC m=+59.074184251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:28.409694 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:28.409587    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:36.409578262 +0000 UTC m=+65.074203483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:29.478944 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:29.478917    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-856bp_1796f663-d2a4-4f37-9442-3ea4ead2efc4/dns-node-resolver/0.log"
Jul 27 21:05:30.279513 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:30.279490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hf45z_8e468562-5b4a-418b-8ae4-360be809fde2/node-ca/0.log"
Jul 27 21:05:30.425499 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:30.425468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:30.425631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:30.425530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:30.425631 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:30.425595    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:30.425631 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:30.425621    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:30.425741 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:30.425679    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:34.425661043 +0000 UTC m=+63.090286260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:30.425741 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:30.425697    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:34.425688811 +0000 UTC m=+63.090314028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:31.278816 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:31.278789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6skq5_a6506781-41f2-4f90-ad2b-a0d7f9ccc627/serve-healthcheck-canary/0.log"
Jul 27 21:05:32.202706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:32.202677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk54c"
Jul 27 21:05:34.459743 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:34.459709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:34.460197 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:34.459779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:34.460197 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:34.459846    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:34.460197 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:34.459868    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:34.460197 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:34.459910    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.459894851 +0000 UTC m=+71.124520066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:34.460197 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:34.459925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.459919713 +0000 UTC m=+71.124544928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:36.274625 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:36.274586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:36.275052 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:36.274727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:36.275052 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.274741    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:52.274723661 +0000 UTC m=+80.939348877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:36.275052 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.274823    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:36.275052 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.274880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs podName:ff98cefc-9b31-4473-bd21-68d1dc2b191d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:52.274864275 +0000 UTC m=+80.939489493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs") pod "router-default-cb9c7cff9-gn874" (UID: "ff98cefc-9b31-4473-bd21-68d1dc2b191d") : secret "router-metrics-certs-default" not found
Jul 27 21:05:36.476811 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:36.476763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:36.476992 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.476887    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 21:05:36.476992 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.476897    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff6b98d77-zvbvf: secret "image-registry-tls" not found
Jul 27 21:05:36.476992 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:36.476947    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls podName:373d73b8-9a63-45d7-89f2-a67078a6de9d nodeName:}" failed. No retries permitted until 2026-07-27 21:05:52.47693214 +0000 UTC m=+81.141557356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls") pod "image-registry-ff6b98d77-zvbvf" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d") : secret "image-registry-tls" not found
Jul 27 21:05:37.485479 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.485442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:37.485479 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.485482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:05:37.488096 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.488075    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:37.488317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.488301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:37.496135 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:37.496116    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 21:05:37.496245 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:37.496180    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs podName:9a84e1cd-17c3-43dc-b4d7-47c93e63686f nodeName:}" failed. No retries permitted until 2026-07-27 21:06:41.496160207 +0000 UTC m=+130.160785426 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs") pod "network-metrics-daemon-phzrj" (UID: "9a84e1cd-17c3-43dc-b4d7-47c93e63686f") : secret "metrics-daemon-secret" not found
Jul 27 21:05:37.498477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.498455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2eec02d3-3654-484f-9f70-60fbac97a123-original-pull-secret\") pod \"global-pull-secret-syncer-zhdtx\" (UID: \"2eec02d3-3654-484f-9f70-60fbac97a123\") " pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:37.586194 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.586156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:37.588725 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.588696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjb2\" (UniqueName: \"kubernetes.io/projected/1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb-kube-api-access-xdjb2\") pod \"network-check-target-rcjn8\" (UID: \"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb\") " pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:37.784647 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.784558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zhdtx"
Jul 27 21:05:37.891213 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.891190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p5l9j\""
Jul 27 21:05:37.897219 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.897199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:37.913696 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:37.913675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zhdtx"]
Jul 27 21:05:37.917517 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:37.917494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eec02d3_3654_484f_9f70_60fbac97a123.slice/crio-86e69d30dede99880464f5b6839e429695a2da49dfa8a2013dd1fc9a5e35ca48 WatchSource:0}: Error finding container 86e69d30dede99880464f5b6839e429695a2da49dfa8a2013dd1fc9a5e35ca48: Status 404 returned error can't find the container with id 86e69d30dede99880464f5b6839e429695a2da49dfa8a2013dd1fc9a5e35ca48
Jul 27 21:05:38.019708 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.019669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rcjn8"]
Jul 27 21:05:38.022060 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:38.022029    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea324c2_61c8_4c0d_af90_6f1aa62a2fcb.slice/crio-fc014ebc9aa4ab758d1279045c8f75415774849c1ed3682716580477ee91a598 WatchSource:0}: Error finding container fc014ebc9aa4ab758d1279045c8f75415774849c1ed3682716580477ee91a598: Status 404 returned error can't find the container with id fc014ebc9aa4ab758d1279045c8f75415774849c1ed3682716580477ee91a598
Jul 27 21:05:38.303380 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.303297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rcjn8" event={"ID":"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb","Type":"ContainerStarted","Data":"1fd7218360c248cde95890e8cb8b58efced47788c2b0a01aa04ad36f91c31d6c"}
Jul 27 21:05:38.303380 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.303341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rcjn8" event={"ID":"1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb","Type":"ContainerStarted","Data":"fc014ebc9aa4ab758d1279045c8f75415774849c1ed3682716580477ee91a598"}
Jul 27 21:05:38.303594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.303384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:05:38.304504 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.304484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhdtx" event={"ID":"2eec02d3-3654-484f-9f70-60fbac97a123","Type":"ContainerStarted","Data":"86e69d30dede99880464f5b6839e429695a2da49dfa8a2013dd1fc9a5e35ca48"}
Jul 27 21:05:38.334324 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.334270    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rcjn8" podStartSLOduration=66.334236613 podStartE2EDuration="1m6.334236613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:38.333298404 +0000 UTC m=+66.997923662" watchObservedRunningTime="2026-07-27 21:05:38.334236613 +0000 UTC m=+66.998861868"
Jul 27 21:05:38.596365 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.596326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:38.598994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.598971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ae68659-399b-4891-a9f8-bf55fae841a8-metrics-tls\") pod \"dns-default-bmpfv\" (UID: \"9ae68659-399b-4891-a9f8-bf55fae841a8\") " pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:38.818981 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.818947    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k6b4w\""
Jul 27 21:05:38.826162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.826124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:38.974149 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:38.974123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmpfv"]
Jul 27 21:05:38.976358 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:38.976330    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae68659_399b_4891_a9f8_bf55fae841a8.slice/crio-8a636ac6b28fae0f88bdbaf33df70ccf436ce4bac8f6f855edc14c925263275e WatchSource:0}: Error finding container 8a636ac6b28fae0f88bdbaf33df70ccf436ce4bac8f6f855edc14c925263275e: Status 404 returned error can't find the container with id 8a636ac6b28fae0f88bdbaf33df70ccf436ce4bac8f6f855edc14c925263275e
Jul 27 21:05:39.312898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:39.312861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmpfv" event={"ID":"9ae68659-399b-4891-a9f8-bf55fae841a8","Type":"ContainerStarted","Data":"8a636ac6b28fae0f88bdbaf33df70ccf436ce4bac8f6f855edc14c925263275e"}
Jul 27 21:05:42.328312 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.328041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zhdtx" event={"ID":"2eec02d3-3654-484f-9f70-60fbac97a123","Type":"ContainerStarted","Data":"4dfb565ba9e95487a0801958b00492bc5f389d4fc2fe062033f1abf7bc40dcdb"}
Jul 27 21:05:42.329881 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.329853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmpfv" event={"ID":"9ae68659-399b-4891-a9f8-bf55fae841a8","Type":"ContainerStarted","Data":"0facdcb0b73b1ceb0bfb785225709209a88ad5dc5c9d51efcc98b07a628c9469"}
Jul 27 21:05:42.360036 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.359984    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zhdtx" podStartSLOduration=66.102588665 podStartE2EDuration="1m10.359969338s" totalImagesPullingTime="4.257380673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:42.359715883 +0000 UTC m=+71.024341122" watchObservedRunningTime="2026-07-27 21:05:42.359969338 +0000 UTC m=+71.024594576"
Jul 27 21:05:42.527445 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.527377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:42.527601 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.527445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:42.527601 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:42.527552    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:42.527710 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:42.527626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls podName:445ace8f-0db8-449f-916a-0daaa96c9691 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:58.527605774 +0000 UTC m=+87.192230991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxbp" (UID: "445ace8f-0db8-449f-916a-0daaa96c9691") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:42.527861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:42.527839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/445ace8f-0db8-449f-916a-0daaa96c9691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:43.333931 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:43.333891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmpfv" event={"ID":"9ae68659-399b-4891-a9f8-bf55fae841a8","Type":"ContainerStarted","Data":"e4a05482fea6ef9ed4bdf4be9c0a0b4d3540ea24a37d74f2917a676707ab2627"}
Jul 27 21:05:43.351661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:43.351604    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bmpfv" podStartSLOduration=34.1584356 podStartE2EDuration="37.351589682s" totalImagesPullingTime="3.193154082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:43.350805379 +0000 UTC m=+72.015430617" watchObservedRunningTime="2026-07-27 21:05:43.351589682 +0000 UTC m=+72.016214897"
Jul 27 21:05:44.337326 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:44.337287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:52.297111 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.296971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:52.297111 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.297025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:52.297624 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.297607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff98cefc-9b31-4473-bd21-68d1dc2b191d-service-ca-bundle\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:52.299453 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.299430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff98cefc-9b31-4473-bd21-68d1dc2b191d-metrics-certs\") pod \"router-default-cb9c7cff9-gn874\" (UID: \"ff98cefc-9b31-4473-bd21-68d1dc2b191d\") " pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:52.482477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.482445    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff6b98d77-zvbvf"]
Jul 27 21:05:52.482656 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:52.482638    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf" podUID="373d73b8-9a63-45d7-89f2-a67078a6de9d"
Jul 27 21:05:52.498683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.498657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:52.501084 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.501064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"image-registry-ff6b98d77-zvbvf\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") " pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:52.540781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.540758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pj45f\""
Jul 27 21:05:52.548947 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.548896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:52.577929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.577903    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-zv4gm"]
Jul 27 21:05:52.580331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.580312    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"]
Jul 27 21:05:52.580464 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.580447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:05:52.582144 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.582125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.586304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.585921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 21:05:52.586304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.586175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xxd42\""
Jul 27 21:05:52.586477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.586359    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 21:05:52.602451 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.602362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-zv4gm"]
Jul 27 21:05:52.603818 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.603793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"]
Jul 27 21:05:52.698297 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.698267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cb9c7cff9-gn874"]
Jul 27 21:05:52.701440 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-installation-pull-secrets\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c034b8cc-a392-4070-9c43-679e7ac36d59-ca-trust-extracted\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pspr\" (UniqueName: \"kubernetes.io/projected/7a5f8ff0-9f6a-463b-b704-22086fd8b8da-kube-api-access-2pspr\") pod \"downloads-7c97dc4555-zv4gm\" (UID: \"7a5f8ff0-9f6a-463b-b704-22086fd8b8da\") " pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:05:52.701553 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:52.701513    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff98cefc_9b31_4473_bd21_68d1dc2b191d.slice/crio-403984f8401f0fe5a24dc987089ad973a1b4beff0abdad017e6a25a1155e12d4 WatchSource:0}: Error finding container 403984f8401f0fe5a24dc987089ad973a1b4beff0abdad017e6a25a1155e12d4: Status 404 returned error can't find the container with id 403984f8401f0fe5a24dc987089ad973a1b4beff0abdad017e6a25a1155e12d4
Jul 27 21:05:52.701553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-tls\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-image-registry-private-configuration\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mld2j\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-kube-api-access-mld2j\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-bound-sa-token\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-trusted-ca\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.701850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.701758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-certificates\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.802901 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-trusted-ca\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.802901 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-certificates\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-installation-pull-secrets\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c034b8cc-a392-4070-9c43-679e7ac36d59-ca-trust-extracted\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pspr\" (UniqueName: \"kubernetes.io/projected/7a5f8ff0-9f6a-463b-b704-22086fd8b8da-kube-api-access-2pspr\") pod \"downloads-7c97dc4555-zv4gm\" (UID: \"7a5f8ff0-9f6a-463b-b704-22086fd8b8da\") " pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.802993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-tls\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.803034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-image-registry-private-configuration\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.803058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mld2j\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-kube-api-access-mld2j\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.803085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-bound-sa-token\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.803704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.803680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c034b8cc-a392-4070-9c43-679e7ac36d59-ca-trust-extracted\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.804009 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.803969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-trusted-ca\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.804103 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.804082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-certificates\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.805610 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.805590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-installation-pull-secrets\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.805700 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.805675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c034b8cc-a392-4070-9c43-679e7ac36d59-image-registry-private-configuration\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.805739 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.805725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-registry-tls\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.824832 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.824805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mld2j\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-kube-api-access-mld2j\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.825116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.825100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c034b8cc-a392-4070-9c43-679e7ac36d59-bound-sa-token\") pod \"image-registry-fb7cf5bd4-w5xsz\" (UID: \"c034b8cc-a392-4070-9c43-679e7ac36d59\") " pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:52.825972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.825953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pspr\" (UniqueName: \"kubernetes.io/projected/7a5f8ff0-9f6a-463b-b704-22086fd8b8da-kube-api-access-2pspr\") pod \"downloads-7c97dc4555-zv4gm\" (UID: \"7a5f8ff0-9f6a-463b-b704-22086fd8b8da\") " pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:05:52.891140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.891121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:05:52.899687 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.899673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d6g8r\""
Jul 27 21:05:52.907803 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:52.907785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:53.072709 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.072609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-zv4gm"]
Jul 27 21:05:53.075082 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:53.075055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5f8ff0_9f6a_463b_b704_22086fd8b8da.slice/crio-44d694e17f6e5d6762c7ca0d33c2aa0cf60ec3c4234b9907ddf081725b58398f WatchSource:0}: Error finding container 44d694e17f6e5d6762c7ca0d33c2aa0cf60ec3c4234b9907ddf081725b58398f: Status 404 returned error can't find the container with id 44d694e17f6e5d6762c7ca0d33c2aa0cf60ec3c4234b9907ddf081725b58398f
Jul 27 21:05:53.113675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.113643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"]
Jul 27 21:05:53.117077 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:53.117057    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc034b8cc_a392_4070_9c43_679e7ac36d59.slice/crio-e59ff175eb7bd078034bd4c81f52aa634169b14462feaa12fd3e6ac51e6d3300 WatchSource:0}: Error finding container e59ff175eb7bd078034bd4c81f52aa634169b14462feaa12fd3e6ac51e6d3300: Status 404 returned error can't find the container with id e59ff175eb7bd078034bd4c81f52aa634169b14462feaa12fd3e6ac51e6d3300
Jul 27 21:05:53.366825 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.366750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-zv4gm" event={"ID":"7a5f8ff0-9f6a-463b-b704-22086fd8b8da","Type":"ContainerStarted","Data":"44d694e17f6e5d6762c7ca0d33c2aa0cf60ec3c4234b9907ddf081725b58398f"}
Jul 27 21:05:53.368383 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.368350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cb9c7cff9-gn874" event={"ID":"ff98cefc-9b31-4473-bd21-68d1dc2b191d","Type":"ContainerStarted","Data":"51b399d81ba5ff135fbadfeff473173c6d9fd4590dcac3304c232a7de33eadef"}
Jul 27 21:05:53.368496 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.368404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cb9c7cff9-gn874" event={"ID":"ff98cefc-9b31-4473-bd21-68d1dc2b191d","Type":"ContainerStarted","Data":"403984f8401f0fe5a24dc987089ad973a1b4beff0abdad017e6a25a1155e12d4"}
Jul 27 21:05:53.369863 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.369838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz" event={"ID":"c034b8cc-a392-4070-9c43-679e7ac36d59","Type":"ContainerStarted","Data":"0f9491add458118beebbba3bb5d193378addd2252bdba4dd7af50d0963782b92"}
Jul 27 21:05:53.369863 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.369854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:53.369979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.369881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz" event={"ID":"c034b8cc-a392-4070-9c43-679e7ac36d59","Type":"ContainerStarted","Data":"e59ff175eb7bd078034bd4c81f52aa634169b14462feaa12fd3e6ac51e6d3300"}
Jul 27 21:05:53.369979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.369962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:05:53.375184 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.375168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:53.389274 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.389214    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-cb9c7cff9-gn874" podStartSLOduration=33.389198218 podStartE2EDuration="33.389198218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:53.388790802 +0000 UTC m=+82.053416042" watchObservedRunningTime="2026-07-27 21:05:53.389198218 +0000 UTC m=+82.053823457"
Jul 27 21:05:53.438070 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.437332    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz" podStartSLOduration=1.437312721 podStartE2EDuration="1.437312721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:53.436698714 +0000 UTC m=+82.101323953" watchObservedRunningTime="2026-07-27 21:05:53.437312721 +0000 UTC m=+82.101937962"
Jul 27 21:05:53.508675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.508831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.508831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.508920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.508920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.509030 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84spd\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.509030 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.508988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.509030 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:05:53.509175 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509027    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration\") pod \"373d73b8-9a63-45d7-89f2-a67078a6de9d\" (UID: \"373d73b8-9a63-45d7-89f2-a67078a6de9d\") "
Jul 27 21:05:53.509175 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509122    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:05:53.509175 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:53.510293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509432    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/373d73b8-9a63-45d7-89f2-a67078a6de9d-ca-trust-extracted\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.510293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509452    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-certificates\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.510293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.509464    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/373d73b8-9a63-45d7-89f2-a67078a6de9d-trusted-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.511840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.511810    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:53.511959 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.511816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:53.511959 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.511866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:05:53.512408 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.512386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:05:53.512543 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.512528    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd" (OuterVolumeSpecName: "kube-api-access-84spd") pod "373d73b8-9a63-45d7-89f2-a67078a6de9d" (UID: "373d73b8-9a63-45d7-89f2-a67078a6de9d"). InnerVolumeSpecName "kube-api-access-84spd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:53.549711 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.549678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:53.552804 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.552781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:53.610055 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.610015    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-installation-pull-secrets\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.610055 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.610054    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/373d73b8-9a63-45d7-89f2-a67078a6de9d-image-registry-private-configuration\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.610330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.610070    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-bound-sa-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.610330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.610085    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-registry-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:53.610330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:53.610098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84spd\" (UniqueName: \"kubernetes.io/projected/373d73b8-9a63-45d7-89f2-a67078a6de9d-kube-api-access-84spd\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:05:54.341929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.341902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bmpfv"
Jul 27 21:05:54.374608 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.374575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff6b98d77-zvbvf"
Jul 27 21:05:54.375178 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.375145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:54.376668 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.376648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-cb9c7cff9-gn874"
Jul 27 21:05:54.456318 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.456083    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff6b98d77-zvbvf"]
Jul 27 21:05:54.461310 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.461277    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-ff6b98d77-zvbvf"]
Jul 27 21:05:54.867821 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.867788    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"]
Jul 27 21:05:54.870344 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.870326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:54.874915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.874893    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 21:05:54.875032 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.874947    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-w88l8\""
Jul 27 21:05:54.938798 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:54.938767    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"]
Jul 27 21:05:55.021799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.021759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6v6g\" (UID: \"0552e627-a250-483e-a98a-91e17d0c0776\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:55.122381 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.122299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6v6g\" (UID: \"0552e627-a250-483e-a98a-91e17d0c0776\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:55.122520 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:55.122469    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:05:55.122579 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:05:55.122568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates podName:0552e627-a250-483e-a98a-91e17d0c0776 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:55.622544833 +0000 UTC m=+84.287170071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-q6v6g" (UID: "0552e627-a250-483e-a98a-91e17d0c0776") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:05:55.626698 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.626663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6v6g\" (UID: \"0552e627-a250-483e-a98a-91e17d0c0776\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:55.629525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.629497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0552e627-a250-483e-a98a-91e17d0c0776-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q6v6g\" (UID: \"0552e627-a250-483e-a98a-91e17d0c0776\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:55.781994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.781841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:55.878235 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.878153    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373d73b8-9a63-45d7-89f2-a67078a6de9d" path="/var/lib/kubelet/pods/373d73b8-9a63-45d7-89f2-a67078a6de9d/volumes"
Jul 27 21:05:55.920352 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:55.920327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"]
Jul 27 21:05:55.923980 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:55.923943    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0552e627_a250_483e_a98a_91e17d0c0776.slice/crio-bb9c6b13c64a157a334cce5c16bc059582316ea9e2e5c4dcc42557d547813e50 WatchSource:0}: Error finding container bb9c6b13c64a157a334cce5c16bc059582316ea9e2e5c4dcc42557d547813e50: Status 404 returned error can't find the container with id bb9c6b13c64a157a334cce5c16bc059582316ea9e2e5c4dcc42557d547813e50
Jul 27 21:05:56.382907 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:56.382871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g" event={"ID":"0552e627-a250-483e-a98a-91e17d0c0776","Type":"ContainerStarted","Data":"bb9c6b13c64a157a334cce5c16bc059582316ea9e2e5c4dcc42557d547813e50"}
Jul 27 21:05:57.388569 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.388529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g" event={"ID":"0552e627-a250-483e-a98a-91e17d0c0776","Type":"ContainerStarted","Data":"8cd61ec78415a08ed724d539a739e7da6441d2861be9081b5de1b00660ff0fc8"}
Jul 27 21:05:57.389046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.388715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:57.393833 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.393813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g"
Jul 27 21:05:57.405707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.405669    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q6v6g" podStartSLOduration=2.360214193 podStartE2EDuration="3.405657673s" totalImagesPullingTime="1.04544348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:57.404915387 +0000 UTC m=+86.069540628" watchObservedRunningTime="2026-07-27 21:05:57.405657673 +0000 UTC m=+86.070282911"
Jul 27 21:05:57.923584 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.923547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"]
Jul 27 21:05:57.925869 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.925851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:57.931783 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.931761    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:05:57.932733 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.932710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7vc49\""
Jul 27 21:05:57.932826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.932767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:05:57.932898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.932713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 21:05:57.933050 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.933034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:05:57.933106 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.933048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 21:05:57.937281 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:57.937242    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"]
Jul 27 21:05:58.046180 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.046139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/055078c0-d43c-49d8-b799-090f0246e0b0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.046180 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.046179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gj7j\" (UniqueName: \"kubernetes.io/projected/055078c0-d43c-49d8-b799-090f0246e0b0-kube-api-access-4gj7j\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.046412 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.046206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.046412 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.046348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.146765 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.146736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.146925 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.146814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/055078c0-d43c-49d8-b799-090f0246e0b0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.146925 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.146840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gj7j\" (UniqueName: \"kubernetes.io/projected/055078c0-d43c-49d8-b799-090f0246e0b0-kube-api-access-4gj7j\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.146925 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.146867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.147657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.147632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/055078c0-d43c-49d8-b799-090f0246e0b0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.149572 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.149546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.149709 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.149690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/055078c0-d43c-49d8-b799-090f0246e0b0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.159785 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.159760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gj7j\" (UniqueName: \"kubernetes.io/projected/055078c0-d43c-49d8-b799-090f0246e0b0-kube-api-access-4gj7j\") pod \"prometheus-operator-84d69f794d-pnpqk\" (UID: \"055078c0-d43c-49d8-b799-090f0246e0b0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.235244 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.235168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"
Jul 27 21:05:58.379339 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.379308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-pnpqk"]
Jul 27 21:05:58.382285 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:58.382244    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055078c0_d43c_49d8_b799_090f0246e0b0.slice/crio-8a1a7de8d0596c0fe39df09780c80f98753ce6bd124e996a0518e5184d533129 WatchSource:0}: Error finding container 8a1a7de8d0596c0fe39df09780c80f98753ce6bd124e996a0518e5184d533129: Status 404 returned error can't find the container with id 8a1a7de8d0596c0fe39df09780c80f98753ce6bd124e996a0518e5184d533129
Jul 27 21:05:58.393657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.393630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk" event={"ID":"055078c0-d43c-49d8-b799-090f0246e0b0","Type":"ContainerStarted","Data":"8a1a7de8d0596c0fe39df09780c80f98753ce6bd124e996a0518e5184d533129"}
Jul 27 21:05:58.548882 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.548809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:58.551329 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.551299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/445ace8f-0db8-449f-916a-0daaa96c9691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxbp\" (UID: \"445ace8f-0db8-449f-916a-0daaa96c9691\") " pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:58.815956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.815925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j726j\""
Jul 27 21:05:58.822952 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.822932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxbp"
Jul 27 21:05:58.997937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:58.997908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxbp"]
Jul 27 21:05:59.001205 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:05:59.001172    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445ace8f_0db8_449f_916a_0daaa96c9691.slice/crio-030972b33c4e3c1ec5055a3b991dfd5c8ab02257ceb891d562845e32c0c8736e WatchSource:0}: Error finding container 030972b33c4e3c1ec5055a3b991dfd5c8ab02257ceb891d562845e32c0c8736e: Status 404 returned error can't find the container with id 030972b33c4e3c1ec5055a3b991dfd5c8ab02257ceb891d562845e32c0c8736e
Jul 27 21:05:59.404275 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:05:59.404219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxbp" event={"ID":"445ace8f-0db8-449f-916a-0daaa96c9691","Type":"ContainerStarted","Data":"030972b33c4e3c1ec5055a3b991dfd5c8ab02257ceb891d562845e32c0c8736e"}
Jul 27 21:06:00.409825 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.409789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk" event={"ID":"055078c0-d43c-49d8-b799-090f0246e0b0","Type":"ContainerStarted","Data":"75bd27e07d5d238399e78e24640fa65932b691fe55127497e298edc247aaa49a"}
Jul 27 21:06:00.767176 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.767129    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:00.769727 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.769705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.772836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.772659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 21:06:00.774484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.773950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 21:06:00.774484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.774162    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 21:06:00.774484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.774174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fljv\""
Jul 27 21:06:00.774484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.774425    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 21:06:00.775854 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.775832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 21:06:00.779476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.779458    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 21:06:00.782074 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.782051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:00.867845 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.867808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.867866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.867902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.867958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.868001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dt8\" (UniqueName: \"kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.868041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.868237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.868081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969068 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969527 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dt8\" (UniqueName: \"kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969852 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.969950 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.969858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.970095 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.970058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.970341 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.970318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.972478 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.972457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.973554 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.973533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:00.979846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:00.979823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dt8\" (UniqueName: \"kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8\") pod \"console-64c555bf77-27zjt\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") " pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:01.084304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:01.084249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:01.248616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:01.248559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:01.252112 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:06:01.252080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ff343f_45f3_4333_a31e_d4f2c8409237.slice/crio-aceee50db5faaf79e6db283bc79b6f6eb7b84662cdda67399a70ffffa32dca83 WatchSource:0}: Error finding container aceee50db5faaf79e6db283bc79b6f6eb7b84662cdda67399a70ffffa32dca83: Status 404 returned error can't find the container with id aceee50db5faaf79e6db283bc79b6f6eb7b84662cdda67399a70ffffa32dca83
Jul 27 21:06:01.417929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:01.417834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk" event={"ID":"055078c0-d43c-49d8-b799-090f0246e0b0","Type":"ContainerStarted","Data":"d859f67adfce4ee5d71d99132e6818e17e44efe3a5f2e5b04960be6d2b1d78d2"}
Jul 27 21:06:01.419986 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:01.419957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c555bf77-27zjt" event={"ID":"82ff343f-45f3-4333-a31e-d4f2c8409237","Type":"ContainerStarted","Data":"aceee50db5faaf79e6db283bc79b6f6eb7b84662cdda67399a70ffffa32dca83"}
Jul 27 21:06:01.435834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:01.435780    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-pnpqk" podStartSLOduration=2.533999387 podStartE2EDuration="4.435758629s" totalImagesPullingTime="1.901759242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:01.435069461 +0000 UTC m=+90.099694719" watchObservedRunningTime="2026-07-27 21:06:01.435758629 +0000 UTC m=+90.100383868"
Jul 27 21:06:03.306077 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.306040    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"]
Jul 27 21:06:03.313168 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.313142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.316069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.316043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:06:03.316069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.316062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 21:06:03.316401 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.316382    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m8xb8\""
Jul 27 21:06:03.321725 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.321701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"]
Jul 27 21:06:03.327828 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.326449    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"]
Jul 27 21:06:03.331214 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.331196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.334893 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.334869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:06:03.335892 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.335330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sr4b7\""
Jul 27 21:06:03.335892 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.335536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 21:06:03.335892 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.335741    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 21:06:03.353681 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.353658    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-286sf"]
Jul 27 21:06:03.360035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.359764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"]
Jul 27 21:06:03.360035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.359884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.363089 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.362789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rsf9d\""
Jul 27 21:06:03.363089 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.362791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:06:03.364716 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.364694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:06:03.365027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.365007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:06:03.390372 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.390243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mf88\" (UniqueName: \"kubernetes.io/projected/d671ec71-956f-47fb-af06-b3f6c5cc302e-kube-api-access-7mf88\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.390878 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.390558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.390878 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.390595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d671ec71-956f-47fb-af06-b3f6c5cc302e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.390878 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.390707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.491178 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.491382 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstjw\" (UniqueName: \"kubernetes.io/projected/41404943-3602-4dc6-8231-c0574cda3c58-kube-api-access-pstjw\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.491453 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.491510 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d671ec71-956f-47fb-af06-b3f6c5cc302e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.491510 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.491619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-wtmp\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.491619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-root\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.491619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.491781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41404943-3602-4dc6-8231-c0574cda3c58-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.491781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.491781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.491781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-metrics-client-ca\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.491781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vv5j\" (UniqueName: \"kubernetes.io/projected/ffd32ae6-a080-40f3-a119-137a013c6d4f-kube-api-access-9vv5j\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-sys\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mf88\" (UniqueName: \"kubernetes.io/projected/d671ec71-956f-47fb-af06-b3f6c5cc302e-kube-api-access-7mf88\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.492035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.491957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-textfile\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.492370 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:03.492093    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 21:06:03.492370 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:03.492144    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls podName:d671ec71-956f-47fb-af06-b3f6c5cc302e nodeName:}" failed. No retries permitted until 2026-07-27 21:06:03.992129123 +0000 UTC m=+92.656754339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-vd965" (UID: "d671ec71-956f-47fb-af06-b3f6c5cc302e") : secret "openshift-state-metrics-tls" not found
Jul 27 21:06:03.493043 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.493008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d671ec71-956f-47fb-af06-b3f6c5cc302e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.495621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.495579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.505152 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.505131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mf88\" (UniqueName: \"kubernetes.io/projected/d671ec71-956f-47fb-af06-b3f6c5cc302e-kube-api-access-7mf88\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:03.592814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-wtmp\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.592814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-root\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.592814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.592814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41404943-3602-4dc6-8231-c0574cda3c58-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.592814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-metrics-client-ca\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vv5j\" (UniqueName: \"kubernetes.io/projected/ffd32ae6-a080-40f3-a119-137a013c6d4f-kube-api-access-9vv5j\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-sys\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.592952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-textfile\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstjw\" (UniqueName: \"kubernetes.io/projected/41404943-3602-4dc6-8231-c0574cda3c58-kube-api-access-pstjw\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.593221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593778 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:03.593696    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 21:06:03.593778 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.593717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-wtmp\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.593778 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:03.593772    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls podName:ffd32ae6-a080-40f3-a119-137a013c6d4f nodeName:}" failed. No retries permitted until 2026-07-27 21:06:04.093753096 +0000 UTC m=+92.758378316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls") pod "node-exporter-286sf" (UID: "ffd32ae6-a080-40f3-a119-137a013c6d4f") : secret "node-exporter-tls" not found
Jul 27 21:06:03.594365 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.594014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-root\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.594365 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.594221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd32ae6-a080-40f3-a119-137a013c6d4f-sys\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.594847 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.594778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.594943 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.594874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.595373 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.595345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.595513 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.595348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd32ae6-a080-40f3-a119-137a013c6d4f-metrics-client-ca\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.595683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.595625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41404943-3602-4dc6-8231-c0574cda3c58-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.595934 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.595907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-textfile\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.597396 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.597372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.599314 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.599290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.599440 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.599416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41404943-3602-4dc6-8231-c0574cda3c58-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.603151 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.603128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vv5j\" (UniqueName: \"kubernetes.io/projected/ffd32ae6-a080-40f3-a119-137a013c6d4f-kube-api-access-9vv5j\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:03.603386 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.603355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstjw\" (UniqueName: \"kubernetes.io/projected/41404943-3602-4dc6-8231-c0574cda3c58-kube-api-access-pstjw\") pod \"kube-state-metrics-56d9f944b4-np7mb\" (UID: \"41404943-3602-4dc6-8231-c0574cda3c58\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.644431 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.644398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"
Jul 27 21:06:03.997632 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:03.997531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:04.000512 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:04.000483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d671ec71-956f-47fb-af06-b3f6c5cc302e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vd965\" (UID: \"d671ec71-956f-47fb-af06-b3f6c5cc302e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:04.098689 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:04.098654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:04.098875 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:04.098847    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 21:06:04.098973 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:06:04.098957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls podName:ffd32ae6-a080-40f3-a119-137a013c6d4f nodeName:}" failed. No retries permitted until 2026-07-27 21:06:05.098933621 +0000 UTC m=+93.763558846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls") pod "node-exporter-286sf" (UID: "ffd32ae6-a080-40f3-a119-137a013c6d4f") : secret "node-exporter-tls" not found
Jul 27 21:06:04.227503 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:04.227471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"
Jul 27 21:06:05.109912 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:05.109872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:05.112784 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:05.112762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ffd32ae6-a080-40f3-a119-137a013c6d4f-node-exporter-tls\") pod \"node-exporter-286sf\" (UID: \"ffd32ae6-a080-40f3-a119-137a013c6d4f\") " pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:05.172049 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:05.172012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-286sf"
Jul 27 21:06:06.340243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.339828    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c5d496fb6-v855k"]
Jul 27 21:06:06.345087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.345061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.347923 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.347881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 21:06:06.348046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348026    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fdn8a6cenaicu\""
Jul 27 21:06:06.348121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348030    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 21:06:06.348121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 21:06:06.348121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348098    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 21:06:06.348121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 21:06:06.348317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.348031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-c5dsn\""
Jul 27 21:06:06.355067 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.355045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5d496fb6-v855k"]
Jul 27 21:06:06.523469 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dns6h\" (UniqueName: \"kubernetes.io/projected/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-kube-api-access-dns6h\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523644 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523821 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-metrics-client-ca\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523821 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.523821 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.523722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-grpc-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-grpc-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dns6h\" (UniqueName: \"kubernetes.io/projected/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-kube-api-access-dns6h\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624988 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-metrics-client-ca\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.624988 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.624820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.625867 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.625812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-metrics-client-ca\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.627660 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.627613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.627972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.627944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.628129 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.628111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.628205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.628181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.628328 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.628305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.628556 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.628537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-secret-grpc-tls\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.635897 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.635875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dns6h\" (UniqueName: \"kubernetes.io/projected/c43bffdf-7834-4311-8bcd-7c1fe7ac8943-kube-api-access-dns6h\") pod \"thanos-querier-c5d496fb6-v855k\" (UID: \"c43bffdf-7834-4311-8bcd-7c1fe7ac8943\") " pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:06.657092 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:06.656895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:09.316200 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:09.316169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rcjn8"
Jul 27 21:06:10.380624 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:10.380540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:13.908639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:13.908584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb"]
Jul 27 21:06:13.929141 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:13.929110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965"]
Jul 27 21:06:13.932308 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:06:13.932276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd671ec71_956f_47fb_af06_b3f6c5cc302e.slice/crio-d3f48b5b3573a2e758708ce823190b204c2d1f5566b5dc9a4f19fce080be43bb WatchSource:0}: Error finding container d3f48b5b3573a2e758708ce823190b204c2d1f5566b5dc9a4f19fce080be43bb: Status 404 returned error can't find the container with id d3f48b5b3573a2e758708ce823190b204c2d1f5566b5dc9a4f19fce080be43bb
Jul 27 21:06:13.961058 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:13.960980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5d496fb6-v855k"]
Jul 27 21:06:13.964126 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:06:13.964093    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43bffdf_7834_4311_8bcd_7c1fe7ac8943.slice/crio-ef58a8ac033db8d5baa6ef2ede38b6c308b6d207c77bdc80a0f2c463c9915eac WatchSource:0}: Error finding container ef58a8ac033db8d5baa6ef2ede38b6c308b6d207c77bdc80a0f2c463c9915eac: Status 404 returned error can't find the container with id ef58a8ac033db8d5baa6ef2ede38b6c308b6d207c77bdc80a0f2c463c9915eac
Jul 27 21:06:14.381764 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.380845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb7cf5bd4-w5xsz"
Jul 27 21:06:14.479890 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.478968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"ef58a8ac033db8d5baa6ef2ede38b6c308b6d207c77bdc80a0f2c463c9915eac"}
Jul 27 21:06:14.480658 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.480605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb" event={"ID":"41404943-3602-4dc6-8231-c0574cda3c58","Type":"ContainerStarted","Data":"6e24e658a1fbdf08bf2eeef3657c993d1d9eb38b620c5784d99ad8abdb6f74a0"}
Jul 27 21:06:14.482571 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.482490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxbp" event={"ID":"445ace8f-0db8-449f-916a-0daaa96c9691","Type":"ContainerStarted","Data":"d17b05617e50129eb1706b001c05725c2bf583d18993918d563340fd67b0e652"}
Jul 27 21:06:14.490039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.489986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c555bf77-27zjt" event={"ID":"82ff343f-45f3-4333-a31e-d4f2c8409237","Type":"ContainerStarted","Data":"30901ab5f22799421aa293ab940746e909b5ddff137e04e7f261ce8c79ceb019"}
Jul 27 21:06:14.494331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.493936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965" event={"ID":"d671ec71-956f-47fb-af06-b3f6c5cc302e","Type":"ContainerStarted","Data":"4ed066f549c663ad7f029ab6f6f5b9347648a697dec65317d07ec441c5a14092"}
Jul 27 21:06:14.494331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.493989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965" event={"ID":"d671ec71-956f-47fb-af06-b3f6c5cc302e","Type":"ContainerStarted","Data":"3324c66d6c236d4352bd16f34401db488c4dbdd2b047963e6fa4444139d12ec3"}
Jul 27 21:06:14.494331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.494019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965" event={"ID":"d671ec71-956f-47fb-af06-b3f6c5cc302e","Type":"ContainerStarted","Data":"d3f48b5b3573a2e758708ce823190b204c2d1f5566b5dc9a4f19fce080be43bb"}
Jul 27 21:06:14.499635 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.497290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-zv4gm" event={"ID":"7a5f8ff0-9f6a-463b-b704-22086fd8b8da","Type":"ContainerStarted","Data":"b4fa3e9e8b191e6faac5df1182472554757517d3551aa47cd2dbce69e6639a7a"}
Jul 27 21:06:14.499974 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.499822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:06:14.503326 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.503250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-286sf" event={"ID":"ffd32ae6-a080-40f3-a119-137a013c6d4f","Type":"ContainerStarted","Data":"6929790bea45459ac1726dfd890bee46c03853fbd95754d98732424e28731ac8"}
Jul 27 21:06:14.512019 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.511966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-zv4gm"
Jul 27 21:06:14.522334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.522278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64c555bf77-27zjt" podStartSLOduration=2.047103688 podStartE2EDuration="14.522240801s" totalImagesPullingTime="12.475137113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:14.519390313 +0000 UTC m=+103.184015573" watchObservedRunningTime="2026-07-27 21:06:14.522240801 +0000 UTC m=+103.186866041"
Jul 27 21:06:14.558504 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:14.557580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-zv4gm" podStartSLOduration=1.852125429 podStartE2EDuration="22.55755653s" totalImagesPullingTime="20.705431101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:14.552829218 +0000 UTC m=+103.217454482" watchObservedRunningTime="2026-07-27 21:06:14.55755653 +0000 UTC m=+103.222181769"
Jul 27 21:06:21.085046 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:21.085004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:22.549916 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.549801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"a25e87317a9feb254d49c9b307000a5fc06cc3ebd1ee9e9ca4ef354a78bd1614"}
Jul 27 21:06:22.550481 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.549938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"c2ad6f896eb3d9f7925c2fa23407f1573262d4adf607bd731b48961f6d781bcf"}
Jul 27 21:06:22.550481 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.549986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"2a23e436461c2b00aaeae8e581710b7b1c183a3e5a99efd6bef7fea9ef58f9ef"}
Jul 27 21:06:22.555826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.555790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb" event={"ID":"41404943-3602-4dc6-8231-c0574cda3c58","Type":"ContainerStarted","Data":"551e294fe1f9062ec20ed8861cb64b4a5fc157920bba2008fe98cac6196d9bae"}
Jul 27 21:06:22.555935 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.555859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb" event={"ID":"41404943-3602-4dc6-8231-c0574cda3c58","Type":"ContainerStarted","Data":"231e8b1bcd6415749fd496bb24301c037eaf789e289e17c156f18a95707fc31a"}
Jul 27 21:06:22.555935 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.555889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb" event={"ID":"41404943-3602-4dc6-8231-c0574cda3c58","Type":"ContainerStarted","Data":"fabc8b6a150e31cc1a26def26b30463df5f48021d9b9a667701fd998fb6b84bd"}
Jul 27 21:06:22.561522 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.561447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxbp" event={"ID":"445ace8f-0db8-449f-916a-0daaa96c9691","Type":"ContainerStarted","Data":"6b429f3183e021ae6dce529d733b3f2a65217da7fd6e46a163bcbea6642ee1e0"}
Jul 27 21:06:22.565592 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.565567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965" event={"ID":"d671ec71-956f-47fb-af06-b3f6c5cc302e","Type":"ContainerStarted","Data":"0bb9cef0bf270a9117c455c14236c6724eb49e3cf44c32ea7a60e18fa642b2ba"}
Jul 27 21:06:22.569169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.568485    2580 generic.go:356] "Generic (PLEG): container finished" podID="ffd32ae6-a080-40f3-a119-137a013c6d4f" containerID="e4b6bf7ee1c3770aa6816402fe2af31682ed38586de2f3ee845d08173bfac9ad" exitCode=0
Jul 27 21:06:22.569169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.568523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-286sf" event={"ID":"ffd32ae6-a080-40f3-a119-137a013c6d4f","Type":"ContainerDied","Data":"e4b6bf7ee1c3770aa6816402fe2af31682ed38586de2f3ee845d08173bfac9ad"}
Jul 27 21:06:22.577251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.577201    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-np7mb" podStartSLOduration=12.023774287 podStartE2EDuration="19.577183134s" totalImagesPullingTime="7.553408847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:22.575928461 +0000 UTC m=+111.240553725" watchObservedRunningTime="2026-07-27 21:06:22.577183134 +0000 UTC m=+111.241808374"
Jul 27 21:06:22.615841 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:22.615796    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vd965" podStartSLOduration=12.276239773 podStartE2EDuration="19.615782614s" totalImagesPullingTime="7.339542841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:22.613686753 +0000 UTC m=+111.278312108" watchObservedRunningTime="2026-07-27 21:06:22.615782614 +0000 UTC m=+111.280407852"
Jul 27 21:06:23.582101 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:23.581961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"35c7b6b2cb2b0c3a918f9723d5bf9a6841326b36dcd3af8c155b827760bc461b"}
Jul 27 21:06:23.582101 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:23.582049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"ec24aa050f5ebb30e3616537a71c1700a36138ef428ff59654caa9bea459fd2b"}
Jul 27 21:06:23.585289 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:23.585239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-286sf" event={"ID":"ffd32ae6-a080-40f3-a119-137a013c6d4f","Type":"ContainerStarted","Data":"e46533010604263723a5fe702ad719492acf585f8a95790dea54c2c498215789"}
Jul 27 21:06:23.585395 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:23.585313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-286sf" event={"ID":"ffd32ae6-a080-40f3-a119-137a013c6d4f","Type":"ContainerStarted","Data":"646ce365148f1f6e6dfc8056863834a58d31a5c54b1a21b4ed6357d129a7d11e"}
Jul 27 21:06:23.613370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:23.613325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-286sf" podStartSLOduration=13.469065116 podStartE2EDuration="20.613309225s" totalImagesPullingTime="7.000004322s" totalInitContainerRuntime="144.239787ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:23.612321249 +0000 UTC m=+112.276946508" watchObservedRunningTime="2026-07-27 21:06:23.613309225 +0000 UTC m=+112.277934463"
Jul 27 21:06:24.593234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:24.593194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" event={"ID":"c43bffdf-7834-4311-8bcd-7c1fe7ac8943","Type":"ContainerStarted","Data":"35b0783267149f768e50e3437c4ce6437332e7680364088a58d2c4b87819b3ab"}
Jul 27 21:06:24.658499 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:24.658429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k" podStartSLOduration=10.373163853 podStartE2EDuration="18.65840554s" totalImagesPullingTime="8.285241687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:24.6580753 +0000 UTC m=+113.322700560" watchObservedRunningTime="2026-07-27 21:06:24.65840554 +0000 UTC m=+113.323030779"
Jul 27 21:06:25.599269 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:25.599229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:26.607614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:26.607586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c5d496fb6-v855k"
Jul 27 21:06:27.608524 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:27.608483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxbp" event={"ID":"445ace8f-0db8-449f-916a-0daaa96c9691","Type":"ContainerStarted","Data":"6f36d3348be0521f5a8824c2cb50b05eaa9c6474025108e9e39295c5002fb9e1"}
Jul 27 21:06:27.677533 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:27.677465    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mxbp" podStartSLOduration=34.530939094 podStartE2EDuration="1m1.677440385s" totalImagesPullingTime="27.146501291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:27.67594542 +0000 UTC m=+116.340570659" watchObservedRunningTime="2026-07-27 21:06:27.677440385 +0000 UTC m=+116.342065625"
Jul 27 21:06:39.524491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.524443    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64c555bf77-27zjt" podUID="82ff343f-45f3-4333-a31e-d4f2c8409237" containerName="console" containerID="cri-o://30901ab5f22799421aa293ab940746e909b5ddff137e04e7f261ce8c79ceb019" gracePeriod=15
Jul 27 21:06:39.656617 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.656585    2580 generic.go:356] "Generic (PLEG): container finished" podID="82ff343f-45f3-4333-a31e-d4f2c8409237" containerID="30901ab5f22799421aa293ab940746e909b5ddff137e04e7f261ce8c79ceb019" exitCode=0
Jul 27 21:06:39.656748 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.656666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c555bf77-27zjt" event={"ID":"82ff343f-45f3-4333-a31e-d4f2c8409237","Type":"ContainerDied","Data":"30901ab5f22799421aa293ab940746e909b5ddff137e04e7f261ce8c79ceb019"}
Jul 27 21:06:39.790415 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.790354    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:39.948657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.948657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.948909 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9dt8\" (UniqueName: \"kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.948909 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.948909 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.949067 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.949067 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.948973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert\") pod \"82ff343f-45f3-4333-a31e-d4f2c8409237\" (UID: \"82ff343f-45f3-4333-a31e-d4f2c8409237\") "
Jul 27 21:06:39.949177 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config" (OuterVolumeSpecName: "console-config") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:39.949177 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca" (OuterVolumeSpecName: "service-ca") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:39.949388 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949364    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-console-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:39.949670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949393    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:39.949670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:39.949670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.949469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:39.951221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.951193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:39.951221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.951204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8" (OuterVolumeSpecName: "kube-api-access-j9dt8") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "kube-api-access-j9dt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:39.951369 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:39.951287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82ff343f-45f3-4333-a31e-d4f2c8409237" (UID: "82ff343f-45f3-4333-a31e-d4f2c8409237"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:40.049788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.049726    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-oauth-serving-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:40.049788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.049751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dt8\" (UniqueName: \"kubernetes.io/projected/82ff343f-45f3-4333-a31e-d4f2c8409237-kube-api-access-j9dt8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:40.049788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.049764    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ff343f-45f3-4333-a31e-d4f2c8409237-trusted-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:40.049788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.049778    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-serving-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:40.050004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.049790    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82ff343f-45f3-4333-a31e-d4f2c8409237-console-oauth-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:06:40.661675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.661647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c555bf77-27zjt"
Jul 27 21:06:40.662187 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.661647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c555bf77-27zjt" event={"ID":"82ff343f-45f3-4333-a31e-d4f2c8409237","Type":"ContainerDied","Data":"aceee50db5faaf79e6db283bc79b6f6eb7b84662cdda67399a70ffffa32dca83"}
Jul 27 21:06:40.662187 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.661790    2580 scope.go:117] "RemoveContainer" containerID="30901ab5f22799421aa293ab940746e909b5ddff137e04e7f261ce8c79ceb019"
Jul 27 21:06:40.683778 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.683758    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:40.687766 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:40.687748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64c555bf77-27zjt"]
Jul 27 21:06:41.560509 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.560479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:06:41.562847 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.562829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a84e1cd-17c3-43dc-b4d7-47c93e63686f-metrics-certs\") pod \"network-metrics-daemon-phzrj\" (UID: \"9a84e1cd-17c3-43dc-b4d7-47c93e63686f\") " pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:06:41.693134 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.693109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r4zfp\""
Jul 27 21:06:41.700552 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.700525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phzrj"
Jul 27 21:06:41.855887 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.855862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-phzrj"]
Jul 27 21:06:41.858471 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:06:41.858442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a84e1cd_17c3_43dc_b4d7_47c93e63686f.slice/crio-bfc4eae22c41e6f7962a4a3493192c6f025b6789c857653a65ffd60c8f976c9e WatchSource:0}: Error finding container bfc4eae22c41e6f7962a4a3493192c6f025b6789c857653a65ffd60c8f976c9e: Status 404 returned error can't find the container with id bfc4eae22c41e6f7962a4a3493192c6f025b6789c857653a65ffd60c8f976c9e
Jul 27 21:06:41.877233 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:41.877207    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ff343f-45f3-4333-a31e-d4f2c8409237" path="/var/lib/kubelet/pods/82ff343f-45f3-4333-a31e-d4f2c8409237/volumes"
Jul 27 21:06:42.675276 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:42.675225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phzrj" event={"ID":"9a84e1cd-17c3-43dc-b4d7-47c93e63686f","Type":"ContainerStarted","Data":"bfc4eae22c41e6f7962a4a3493192c6f025b6789c857653a65ffd60c8f976c9e"}
Jul 27 21:06:43.680221 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:43.680189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phzrj" event={"ID":"9a84e1cd-17c3-43dc-b4d7-47c93e63686f","Type":"ContainerStarted","Data":"bced7a3532f78bb7c026501b73f020b5d102109fb4f32524dff8d9a7925043b8"}
Jul 27 21:06:43.680588 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:43.680242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phzrj" event={"ID":"9a84e1cd-17c3-43dc-b4d7-47c93e63686f","Type":"ContainerStarted","Data":"627707ea62bda131b9f0fcc3df585c487902d2197311242c5b90ca89d1113645"}
Jul 27 21:06:43.698508 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:43.698466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-phzrj" podStartSLOduration=130.193267226 podStartE2EDuration="2m11.698443588s" totalImagesPullingTime="1.505176362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:43.698357125 +0000 UTC m=+132.362982364" watchObservedRunningTime="2026-07-27 21:06:43.698443588 +0000 UTC m=+132.363068825"
Jul 27 21:06:56.730860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.730826    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8ca8d50-1f85-489a-a938-779325f199b1" containerID="7f110cace7ec345f13f2de9bbbdfd1433478b13f5fc971b5c034ee17c3d23505" exitCode=0
Jul 27 21:06:56.731349 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.730902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-wtrtk" event={"ID":"f8ca8d50-1f85-489a-a938-779325f199b1","Type":"ContainerDied","Data":"7f110cace7ec345f13f2de9bbbdfd1433478b13f5fc971b5c034ee17c3d23505"}
Jul 27 21:06:56.731349 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.731232    2580 scope.go:117] "RemoveContainer" containerID="7f110cace7ec345f13f2de9bbbdfd1433478b13f5fc971b5c034ee17c3d23505"
Jul 27 21:06:56.732364 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.732341    2580 generic.go:356] "Generic (PLEG): container finished" podID="cc3dc3a3-3420-49b2-9e28-76219d7f91bb" containerID="3f2281347ef5d21dc59c41b81d2feeed575b97de8a0e9c1ff0042a62f799da05" exitCode=0
Jul 27 21:06:56.732477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.732390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4" event={"ID":"cc3dc3a3-3420-49b2-9e28-76219d7f91bb","Type":"ContainerDied","Data":"3f2281347ef5d21dc59c41b81d2feeed575b97de8a0e9c1ff0042a62f799da05"}
Jul 27 21:06:56.732611 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:56.732600    2580 scope.go:117] "RemoveContainer" containerID="3f2281347ef5d21dc59c41b81d2feeed575b97de8a0e9c1ff0042a62f799da05"
Jul 27 21:06:57.547936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:57.547902    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cb9c7cff9-gn874_ff98cefc-9b31-4473-bd21-68d1dc2b191d/router/0.log"
Jul 27 21:06:57.559962 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:57.559935    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6skq5_a6506781-41f2-4f90-ad2b-a0d7f9ccc627/serve-healthcheck-canary/0.log"
Jul 27 21:06:57.738105 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:57.738069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-wtrtk" event={"ID":"f8ca8d50-1f85-489a-a938-779325f199b1","Type":"ContainerStarted","Data":"cfc93bbc7324f9462514112096a9d88d2272621a7b7755e203bf9e6bf934c47b"}
Jul 27 21:06:57.739830 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:06:57.739809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-pb4s4" event={"ID":"cc3dc3a3-3420-49b2-9e28-76219d7f91bb","Type":"ContainerStarted","Data":"37517a46deb0c6f8c837a6678bec184a4dba51ab7c2fbf373543e1dfe5fa08ac"}
Jul 27 21:09:31.696562 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:31.696536    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:09:38.553402 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.553369    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"]
Jul 27 21:09:38.554815 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.553659    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ff343f-45f3-4333-a31e-d4f2c8409237" containerName="console"
Jul 27 21:09:38.554815 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.553670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ff343f-45f3-4333-a31e-d4f2c8409237" containerName="console"
Jul 27 21:09:38.554815 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.553728    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ff343f-45f3-4333-a31e-d4f2c8409237" containerName="console"
Jul 27 21:09:38.555555 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.555536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.558206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.558184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 21:09:38.559015 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.558999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-562z9\""
Jul 27 21:09:38.559064 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.559013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 21:09:38.566752 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.566733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"]
Jul 27 21:09:38.703871 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.703843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.703871 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.703870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.704051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.703902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5jq\" (UniqueName: \"kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.804422 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.804401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5jq\" (UniqueName: \"kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.804540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.804459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.804540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.804477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.804774 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.804757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.804814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.804780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.813097 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.813072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5jq\" (UniqueName: \"kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.865212 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.865188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:38.988278 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.988227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"]
Jul 27 21:09:38.991155 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:09:38.991120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7c5fbd_71b2_4193_85e2_145a9dd4e498.slice/crio-a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c WatchSource:0}: Error finding container a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c: Status 404 returned error can't find the container with id a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c
Jul 27 21:09:38.993345 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:38.993326    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:09:39.337810 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:39.337774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k" event={"ID":"ec7c5fbd-71b2-4193-85e2-145a9dd4e498","Type":"ContainerStarted","Data":"a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c"}
Jul 27 21:09:44.363078 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:44.362992    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerID="e92e3ae60acca26111f46ab61854a2a7154569ce9bdc3a635cc73aeb8bd1c6e6" exitCode=0
Jul 27 21:09:44.363482 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:44.363082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k" event={"ID":"ec7c5fbd-71b2-4193-85e2-145a9dd4e498","Type":"ContainerDied","Data":"e92e3ae60acca26111f46ab61854a2a7154569ce9bdc3a635cc73aeb8bd1c6e6"}
Jul 27 21:09:47.376217 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:47.376182    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerID="7d1cdadae0c78441ec7df394f266c78fffcc5e46aac4b5f8bac02200a4ad0168" exitCode=0
Jul 27 21:09:47.376621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:47.376286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k" event={"ID":"ec7c5fbd-71b2-4193-85e2-145a9dd4e498","Type":"ContainerDied","Data":"7d1cdadae0c78441ec7df394f266c78fffcc5e46aac4b5f8bac02200a4ad0168"}
Jul 27 21:09:54.409211 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:54.409179    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerID="c1eaa0fa26cd994a014706b0fec30b2fd767115749b4876e59c07a29c645db02" exitCode=0
Jul 27 21:09:54.409614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:54.409280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k" event={"ID":"ec7c5fbd-71b2-4193-85e2-145a9dd4e498","Type":"ContainerDied","Data":"c1eaa0fa26cd994a014706b0fec30b2fd767115749b4876e59c07a29c645db02"}
Jul 27 21:09:55.536592 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.536568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:55.550218 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.550197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl5jq\" (UniqueName: \"kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq\") pod \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") "
Jul 27 21:09:55.550302 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.550286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util\") pod \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") "
Jul 27 21:09:55.550364 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.550325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle\") pod \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\" (UID: \"ec7c5fbd-71b2-4193-85e2-145a9dd4e498\") "
Jul 27 21:09:55.550917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.550896    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle" (OuterVolumeSpecName: "bundle") pod "ec7c5fbd-71b2-4193-85e2-145a9dd4e498" (UID: "ec7c5fbd-71b2-4193-85e2-145a9dd4e498"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:55.552464 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.552445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq" (OuterVolumeSpecName: "kube-api-access-rl5jq") pod "ec7c5fbd-71b2-4193-85e2-145a9dd4e498" (UID: "ec7c5fbd-71b2-4193-85e2-145a9dd4e498"). InnerVolumeSpecName "kube-api-access-rl5jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:09:55.555876 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.555848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util" (OuterVolumeSpecName: "util") pod "ec7c5fbd-71b2-4193-85e2-145a9dd4e498" (UID: "ec7c5fbd-71b2-4193-85e2-145a9dd4e498"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:55.651578 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.651539    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:09:55.651578 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.651573    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:09:55.651578 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:55.651584    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5jq\" (UniqueName: \"kubernetes.io/projected/ec7c5fbd-71b2-4193-85e2-145a9dd4e498-kube-api-access-rl5jq\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:09:56.419498 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:56.419456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k" event={"ID":"ec7c5fbd-71b2-4193-85e2-145a9dd4e498","Type":"ContainerDied","Data":"a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c"}
Jul 27 21:09:56.419498 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:56.419482    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5p6h6k"
Jul 27 21:09:56.419498 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:56.419505    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91c0767e6d9e6860c8430b9ee84cbc040d85a3356e5dff17bfbf8bebc295c0c"
Jul 27 21:09:59.516548 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"]
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516854    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="extract"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516865    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="extract"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516882    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="util"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516888    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="util"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516895    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="pull"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516901    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="pull"
Jul 27 21:09:59.516946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.516948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7c5fbd-71b2-4193-85e2-145a9dd4e498" containerName="extract"
Jul 27 21:09:59.520436 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.520421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.522925 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.522904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-w5csq\""
Jul 27 21:09:59.524061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.524043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 27 21:09:59.524149 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.524082    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 27 21:09:59.524354 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.524336    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 27 21:09:59.551309 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.551290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"]
Jul 27 21:09:59.585779 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.585752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq62t\" (UniqueName: \"kubernetes.io/projected/976c0f7d-619e-48a2-a4f6-83a67be341c9-kube-api-access-kq62t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.585891 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.585797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/976c0f7d-619e-48a2-a4f6-83a67be341c9-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.686731 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.686699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/976c0f7d-619e-48a2-a4f6-83a67be341c9-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.686877 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.686772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq62t\" (UniqueName: \"kubernetes.io/projected/976c0f7d-619e-48a2-a4f6-83a67be341c9-kube-api-access-kq62t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.689296 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.689273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/976c0f7d-619e-48a2-a4f6-83a67be341c9-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.700860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.700840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq62t\" (UniqueName: \"kubernetes.io/projected/976c0f7d-619e-48a2-a4f6-83a67be341c9-kube-api-access-kq62t\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-48sdn\" (UID: \"976c0f7d-619e-48a2-a4f6-83a67be341c9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.830594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.830572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:09:59.983582 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:09:59.983544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"]
Jul 27 21:09:59.988815 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:09:59.988787    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976c0f7d_619e_48a2_a4f6_83a67be341c9.slice/crio-ae0ae5c293e9f64092615bfe9cd9911859bc0bcc19e8c25afa0876141b49f606 WatchSource:0}: Error finding container ae0ae5c293e9f64092615bfe9cd9911859bc0bcc19e8c25afa0876141b49f606: Status 404 returned error can't find the container with id ae0ae5c293e9f64092615bfe9cd9911859bc0bcc19e8c25afa0876141b49f606
Jul 27 21:10:00.435624 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:00.435582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn" event={"ID":"976c0f7d-619e-48a2-a4f6-83a67be341c9","Type":"ContainerStarted","Data":"ae0ae5c293e9f64092615bfe9cd9911859bc0bcc19e8c25afa0876141b49f606"}
Jul 27 21:10:04.460131 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:04.460094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn" event={"ID":"976c0f7d-619e-48a2-a4f6-83a67be341c9","Type":"ContainerStarted","Data":"f81dd84192ecdbc50fefe0f392f273dc930ad700fd1dac45651a449ca3bf7808"}
Jul 27 21:10:04.460622 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:04.460158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:10:04.483671 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:04.483483    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn" podStartSLOduration=2.12663094 podStartE2EDuration="5.483463095s" totalImagesPullingTime="3.356832155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:04.481335029 +0000 UTC m=+333.145960269" watchObservedRunningTime="2026-07-27 21:10:04.483463095 +0000 UTC m=+333.148088336"
Jul 27 21:10:05.115331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.115297    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-h9mq2"]
Jul 27 21:10:05.134701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.134677    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.139197 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.139167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-h9mq2"]
Jul 27 21:10:05.141171 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.141144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 27 21:10:05.141609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.141593    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rcthv\""
Jul 27 21:10:05.142087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.142068    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 27 21:10:05.230943 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.230912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-certificates\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.231113 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.230955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2h8\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-kube-api-access-4s2h8\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.331840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.331800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-certificates\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.332608 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.332586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2h8\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-kube-api-access-4s2h8\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.335045 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.335016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-certificates\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.341558 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.341534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2h8\" (UniqueName: \"kubernetes.io/projected/d945a5ef-b2ee-441a-a52c-13ef90c55c46-kube-api-access-4s2h8\") pod \"keda-admission-56f9798bb4-h9mq2\" (UID: \"d945a5ef-b2ee-441a-a52c-13ef90c55c46\") " pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.447754 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.447660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:05.602611 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:05.602581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-h9mq2"]
Jul 27 21:10:05.605343 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:10:05.605319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd945a5ef_b2ee_441a_a52c_13ef90c55c46.slice/crio-153c83ce2429a44c536793d57820d5946fa672faca553ac36aeabb945da23b6d WatchSource:0}: Error finding container 153c83ce2429a44c536793d57820d5946fa672faca553ac36aeabb945da23b6d: Status 404 returned error can't find the container with id 153c83ce2429a44c536793d57820d5946fa672faca553ac36aeabb945da23b6d
Jul 27 21:10:06.470661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:06.470626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2" event={"ID":"d945a5ef-b2ee-441a-a52c-13ef90c55c46","Type":"ContainerStarted","Data":"153c83ce2429a44c536793d57820d5946fa672faca553ac36aeabb945da23b6d"}
Jul 27 21:10:08.480240 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:08.480204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2" event={"ID":"d945a5ef-b2ee-441a-a52c-13ef90c55c46","Type":"ContainerStarted","Data":"f6c42746a4327e159ae5885cfaac226da31572528b27bf190f04964aff5fd098"}
Jul 27 21:10:08.480661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:08.480314    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:10:08.502666 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:08.502614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2" podStartSLOduration=1.2797894300000001 podStartE2EDuration="3.50259703s" totalImagesPullingTime="2.2228076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:08.501512664 +0000 UTC m=+337.166137901" watchObservedRunningTime="2026-07-27 21:10:08.50259703 +0000 UTC m=+337.167222280"
Jul 27 21:10:25.468054 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:25.467970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-48sdn"
Jul 27 21:10:29.486846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:10:29.486821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-h9mq2"
Jul 27 21:11:28.771350 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.771320    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-kxfcc"]
Jul 27 21:11:28.773629 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.773611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.777982 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.777961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-95wp8\""
Jul 27 21:11:28.777982 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.777977    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:11:28.778155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.777988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 21:11:28.778155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.777989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:11:28.784401 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.784380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kxfcc"]
Jul 27 21:11:28.859702 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.859677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftgr\" (UniqueName: \"kubernetes.io/projected/36e7812e-1625-4f99-895f-acb1f45b6e1a-kube-api-access-bftgr\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.859844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.859722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36e7812e-1625-4f99-895f-acb1f45b6e1a-data\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.961139 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.961108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftgr\" (UniqueName: \"kubernetes.io/projected/36e7812e-1625-4f99-895f-acb1f45b6e1a-kube-api-access-bftgr\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.961304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.961161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36e7812e-1625-4f99-895f-acb1f45b6e1a-data\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.961598 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.961582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36e7812e-1625-4f99-895f-acb1f45b6e1a-data\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:28.972937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:28.972908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftgr\" (UniqueName: \"kubernetes.io/projected/36e7812e-1625-4f99-895f-acb1f45b6e1a-kube-api-access-bftgr\") pod \"seaweedfs-64568bcd49-kxfcc\" (UID: \"36e7812e-1625-4f99-895f-acb1f45b6e1a\") " pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:29.082979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:29.082954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:29.214894 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:29.214866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kxfcc"]
Jul 27 21:11:29.216235 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:11:29.216210    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e7812e_1625_4f99_895f_acb1f45b6e1a.slice/crio-c9430abbfcb9f1d96c4fe0a92b38531072e950887858489a4471e25c246ad0b2 WatchSource:0}: Error finding container c9430abbfcb9f1d96c4fe0a92b38531072e950887858489a4471e25c246ad0b2: Status 404 returned error can't find the container with id c9430abbfcb9f1d96c4fe0a92b38531072e950887858489a4471e25c246ad0b2
Jul 27 21:11:29.832047 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:29.831999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kxfcc" event={"ID":"36e7812e-1625-4f99-895f-acb1f45b6e1a","Type":"ContainerStarted","Data":"c9430abbfcb9f1d96c4fe0a92b38531072e950887858489a4471e25c246ad0b2"}
Jul 27 21:11:31.843729 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:31.843689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kxfcc" event={"ID":"36e7812e-1625-4f99-895f-acb1f45b6e1a","Type":"ContainerStarted","Data":"8506fd2e185b224b9f67cb94f88adcc8aba71f94c44dcf257cf3481319197758"}
Jul 27 21:11:31.844176 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:31.843778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:11:31.871073 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:31.871022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-kxfcc" podStartSLOduration=1.442707626 podStartE2EDuration="3.871006835s" totalImagesPullingTime="2.428299209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:31.868904749 +0000 UTC m=+420.533529998" watchObservedRunningTime="2026-07-27 21:11:31.871006835 +0000 UTC m=+420.535632096"
Jul 27 21:11:37.850418 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:11:37.850387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-kxfcc"
Jul 27 21:12:35.922106 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.922076    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d766c59c9-vsgn7"]
Jul 27 21:12:35.924189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.924171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.928681 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.928656    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 21:12:35.928785 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.928701    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 21:12:35.929511 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.929490    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 21:12:35.929773 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.929747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 21:12:35.929890 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.929753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fljv\""
Jul 27 21:12:35.929953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.929905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 21:12:35.936037 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.936021    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 21:12:35.945689 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.945669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d766c59c9-vsgn7"]
Jul 27 21:12:35.982205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-oauth-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982343 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982343 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982343 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-trusted-ca-bundle\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-service-ca\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf654\" (UniqueName: \"kubernetes.io/projected/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-kube-api-access-nf654\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:35.982463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:35.982413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-oauth-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083178 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-trusted-ca-bundle\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083363 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-service-ca\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083363 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf654\" (UniqueName: \"kubernetes.io/projected/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-kube-api-access-nf654\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-oauth-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-oauth-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.083604 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.083477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.084062 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.084014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-service-ca\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.084168 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.084103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-trusted-ca-bundle\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.084168 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.084113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.084248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.084100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-oauth-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.085840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.085819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-oauth-config\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.085964 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.085946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-console-serving-cert\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.094076 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.094059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf654\" (UniqueName: \"kubernetes.io/projected/e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497-kube-api-access-nf654\") pod \"console-7d766c59c9-vsgn7\" (UID: \"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497\") " pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.233886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.233813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:36.369109 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:36.369077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d766c59c9-vsgn7"]
Jul 27 21:12:36.372169 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:12:36.372147    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42b3955_0bb4_4c6d_90c5_a5f9ed3ea497.slice/crio-f1688eb63ad5e9cb627df3b59c3b573068812f22af235ff07ae34eceeead7b44 WatchSource:0}: Error finding container f1688eb63ad5e9cb627df3b59c3b573068812f22af235ff07ae34eceeead7b44: Status 404 returned error can't find the container with id f1688eb63ad5e9cb627df3b59c3b573068812f22af235ff07ae34eceeead7b44
Jul 27 21:12:37.123945 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:37.123907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d766c59c9-vsgn7" event={"ID":"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497","Type":"ContainerStarted","Data":"94b6800181adc3dacb83d1243435a57e05832e7363d4b86f22c81337b73ee008"}
Jul 27 21:12:37.124358 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:37.123989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d766c59c9-vsgn7" event={"ID":"e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497","Type":"ContainerStarted","Data":"f1688eb63ad5e9cb627df3b59c3b573068812f22af235ff07ae34eceeead7b44"}
Jul 27 21:12:37.143653 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:37.143610    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d766c59c9-vsgn7" podStartSLOduration=2.143596736 podStartE2EDuration="2.143596736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:37.141870641 +0000 UTC m=+485.806495880" watchObservedRunningTime="2026-07-27 21:12:37.143596736 +0000 UTC m=+485.808222041"
Jul 27 21:12:46.234837 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.234803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:46.235432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.234864    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:46.239291 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.239248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:46.603879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.603851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-vmkgt"]
Jul 27 21:12:46.606240 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.606220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.611488 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.611468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-68nqw\""
Jul 27 21:12:46.611603 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.611503    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 27 21:12:46.614179 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.614097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pwlgz"]
Jul 27 21:12:46.616356 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.616341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.618795 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.618775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 27 21:12:46.618911 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.618802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4r7zj\""
Jul 27 21:12:46.620099 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.620076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vmkgt"]
Jul 27 21:12:46.631968 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.631950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pwlgz"]
Jul 27 21:12:46.669583 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.669557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-tls-certs\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.669583 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.669584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.669749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.669602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79n4\" (UniqueName: \"kubernetes.io/projected/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-kube-api-access-f79n4\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.669749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.669656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxrn\" (UniqueName: \"kubernetes.io/projected/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-kube-api-access-smxrn\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.770817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.770790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxrn\" (UniqueName: \"kubernetes.io/projected/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-kube-api-access-smxrn\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.770935 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.770853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-tls-certs\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.770935 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.770870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.770935 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.770888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79n4\" (UniqueName: \"kubernetes.io/projected/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-kube-api-access-f79n4\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.771035 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:12:46.771001    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:46.771078 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:12:46.771067    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert podName:157aae4d-5ff7-4198-a0fb-c9f1279e6fef nodeName:}" failed. No retries permitted until 2026-07-27 21:12:47.271048715 +0000 UTC m=+495.935673939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert") pod "odh-model-controller-75d9cdbddf-pwlgz" (UID: "157aae4d-5ff7-4198-a0fb-c9f1279e6fef") : secret "odh-model-controller-webhook-cert" not found
Jul 27 21:12:46.773493 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.773474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-tls-certs\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.781771 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.781751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxrn\" (UniqueName: \"kubernetes.io/projected/c1c06c1c-e051-48f3-b141-4c88ef5eeb61-kube-api-access-smxrn\") pod \"model-serving-api-7b954ffc95-vmkgt\" (UID: \"c1c06c1c-e051-48f3-b141-4c88ef5eeb61\") " pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:46.782237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.782222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79n4\" (UniqueName: \"kubernetes.io/projected/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-kube-api-access-f79n4\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:46.918140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:46.918073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:47.056959 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.056936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vmkgt"]
Jul 27 21:12:47.060711 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:12:47.060679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c06c1c_e051_48f3_b141_4c88ef5eeb61.slice/crio-828bd5894c8c9388fdf04d5af481a6b543fe1c1d81e260b6730ae4484104bf1a WatchSource:0}: Error finding container 828bd5894c8c9388fdf04d5af481a6b543fe1c1d81e260b6730ae4484104bf1a: Status 404 returned error can't find the container with id 828bd5894c8c9388fdf04d5af481a6b543fe1c1d81e260b6730ae4484104bf1a
Jul 27 21:12:47.169524 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.169457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vmkgt" event={"ID":"c1c06c1c-e051-48f3-b141-4c88ef5eeb61","Type":"ContainerStarted","Data":"828bd5894c8c9388fdf04d5af481a6b543fe1c1d81e260b6730ae4484104bf1a"}
Jul 27 21:12:47.173336 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.173317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d766c59c9-vsgn7"
Jul 27 21:12:47.275172 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.275131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:47.277703 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.277672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157aae4d-5ff7-4198-a0fb-c9f1279e6fef-cert\") pod \"odh-model-controller-75d9cdbddf-pwlgz\" (UID: \"157aae4d-5ff7-4198-a0fb-c9f1279e6fef\") " pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:47.526917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.526841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:47.683937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:47.683885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-pwlgz"]
Jul 27 21:12:47.686577 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:12:47.686545    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157aae4d_5ff7_4198_a0fb_c9f1279e6fef.slice/crio-fb1cc6ed7843982476c963b20d68303bb7760462685cfe38cdcbef0e197c6856 WatchSource:0}: Error finding container fb1cc6ed7843982476c963b20d68303bb7760462685cfe38cdcbef0e197c6856: Status 404 returned error can't find the container with id fb1cc6ed7843982476c963b20d68303bb7760462685cfe38cdcbef0e197c6856
Jul 27 21:12:48.175704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:48.175671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz" event={"ID":"157aae4d-5ff7-4198-a0fb-c9f1279e6fef","Type":"ContainerStarted","Data":"fb1cc6ed7843982476c963b20d68303bb7760462685cfe38cdcbef0e197c6856"}
Jul 27 21:12:51.195678 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.195645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz" event={"ID":"157aae4d-5ff7-4198-a0fb-c9f1279e6fef","Type":"ContainerStarted","Data":"d43e15c38a83d4d6c19f70d93c4bbcc59d41315cf08406c3e38526f5355ca176"}
Jul 27 21:12:51.196074 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.195702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:12:51.197137 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.197114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vmkgt" event={"ID":"c1c06c1c-e051-48f3-b141-4c88ef5eeb61","Type":"ContainerStarted","Data":"cbd940c014523909b796ed9601c6d973da5d9c02e9df8a7143c23e65ea7070a5"}
Jul 27 21:12:51.197278 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.197246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:12:51.213917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.213872    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz" podStartSLOduration=2.291132807 podStartE2EDuration="5.213859904s" totalImagesPullingTime="2.922727097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:51.212750704 +0000 UTC m=+499.877375957" watchObservedRunningTime="2026-07-27 21:12:51.213859904 +0000 UTC m=+499.878485295"
Jul 27 21:12:51.230610 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:12:51.230574    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-vmkgt" podStartSLOduration=1.717565874 podStartE2EDuration="5.230562329s" totalImagesPullingTime="3.512996455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:51.229650482 +0000 UTC m=+499.894275722" watchObservedRunningTime="2026-07-27 21:12:51.230562329 +0000 UTC m=+499.895187566"
Jul 27 21:13:02.204452 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:13:02.204421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-pwlgz"
Jul 27 21:13:02.206461 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:13:02.206442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-vmkgt"
Jul 27 21:14:34.707092 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.707058    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:14:34.710688 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.710670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.714983 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.714961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-05e8c-predictor-serving-cert\""
Jul 27 21:14:34.715102 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.715059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:14:34.715205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.715190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-05e8c-kube-rbac-proxy-sar-config\""
Jul 27 21:14:34.715588 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.715574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:14:34.716024 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.716008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:14:34.716118 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.716031    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:14:34.716602 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.716589    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:14:34.732564 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.732541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcbk\" (UniqueName: \"kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.732564 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.732569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.732761 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.732605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.732761 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.732750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.732864 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.732783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.767421 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.767381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:14:34.834062 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834062 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcbk\" (UniqueName: \"kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834319 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:14:34.834244    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-05e8c-predictor-serving-cert" not found
Jul 27 21:14:34.834485 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:14:34.834349    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls podName:29e43173-641a-4b63-8439-0cc93b7379e4 nodeName:}" failed. No retries permitted until 2026-07-27 21:14:35.334327132 +0000 UTC m=+603.998952348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls") pod "isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4") : secret "isvc-raw-sklearn-batcher-05e8c-predictor-serving-cert" not found
Jul 27 21:14:34.834641 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834866 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.834903 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.834861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:34.853482 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:34.853446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcbk\" (UniqueName: \"kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:35.337742 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:35.337709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:35.340313 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:35.340289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:35.620724 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:35.620644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:14:35.754297 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:35.754272    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:14:35.756675 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:14:35.756646    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e43173_641a_4b63_8439_0cc93b7379e4.slice/crio-ec56542498ade1ae2ea49cd23cd85bbc24604ae3355ac37ecc4019af581ca266 WatchSource:0}: Error finding container ec56542498ade1ae2ea49cd23cd85bbc24604ae3355ac37ecc4019af581ca266: Status 404 returned error can't find the container with id ec56542498ade1ae2ea49cd23cd85bbc24604ae3355ac37ecc4019af581ca266
Jul 27 21:14:36.666982 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:36.666942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerStarted","Data":"ec56542498ade1ae2ea49cd23cd85bbc24604ae3355ac37ecc4019af581ca266"}
Jul 27 21:14:39.685902 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:39.685868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerStarted","Data":"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"}
Jul 27 21:14:42.702141 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:42.702059    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e43173-641a-4b63-8439-0cc93b7379e4" containerID="8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e" exitCode=0
Jul 27 21:14:42.702141 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:42.702101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerDied","Data":"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"}
Jul 27 21:14:42.703271 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:42.703241    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:14:56.782687 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:56.782654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerStarted","Data":"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"}
Jul 27 21:14:59.804668 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:14:59.804629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerStarted","Data":"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"}
Jul 27 21:15:02.821478 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:02.821443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerStarted","Data":"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"}
Jul 27 21:15:02.821849 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:02.821650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:15:02.821849 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:02.821696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:15:02.823036 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:02.823004    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:02.841704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:02.841659    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podStartSLOduration=2.9582351669999998 podStartE2EDuration="28.841646901s" totalImagesPullingTime="22.364246375s" totalInitContainerRuntime="3.519165359s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:34 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:02.839415154 +0000 UTC m=+631.504040427" watchObservedRunningTime="2026-07-27 21:15:02.841646901 +0000 UTC m=+631.506272139"
Jul 27 21:15:03.826118 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:03.826082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:03.826578 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:03.826169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:15:03.827028 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:03.827003    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:04.831026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:04.830983    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:04.831483 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:04.831450    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:04.834971 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:04.834952    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:15:05.836105 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:05.836068    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:05.836563 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:05.836385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:15.836478 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:15.836433    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:15.836964 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:15.836918    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:25.836370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:25.836326    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:25.836750 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:25.836662    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:35.836343 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:35.836297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:35.836841 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:35.836816    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:45.836535 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:45.836487    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:45.836943 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:45.836918    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:15:55.836775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:55.836730    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:15:55.837326 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:15:55.837149    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:05.836721 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:05.836670    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:16:05.837148 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:05.837059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:15.836432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:15.836401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:16:15.836871 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:15.836633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:16:29.828292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.828205    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:16:29.828843 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.828576    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" containerID="cri-o://e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474" gracePeriod=30
Jul 27 21:16:29.828932 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.828901    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" containerID="cri-o://678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3" gracePeriod=30
Jul 27 21:16:29.829061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.829017    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" containerID="cri-o://7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c" gracePeriod=30
Jul 27 21:16:29.835913 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.835830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": EOF"
Jul 27 21:16:29.844653 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.844630    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:16:29.848896 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.848880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.851805 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.851784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-6148e-kube-rbac-proxy-sar-config\""
Jul 27 21:16:29.851954 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.851811    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-6148e-predictor-serving-cert\""
Jul 27 21:16:29.864370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.864340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:16:29.892441 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.892417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.892562 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.892459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.892562 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.892535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.892562 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.892554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxss6\" (UniqueName: \"kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.892672 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.892617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.916250 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.916211    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:16:29.919776 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.919757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.922390 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.922362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-6148e-predictor-serving-cert\""
Jul 27 21:16:29.922551 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.922536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-6148e-kube-rbac-proxy-sar-config\""
Jul 27 21:16:29.932084 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.932062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:16:29.993601 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.993700 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.993700 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxss6\" (UniqueName: \"kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.993771 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.993771 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.993860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q4k\" (UniqueName: \"kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.993860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.993860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.993994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:29.993994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.993956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.994088 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.994074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.994088 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:16:29.994084    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-serving-cert: secret "isvc-sklearn-graph-raw-6148e-predictor-serving-cert" not found
Jul 27 21:16:29.994154 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:16:29.994133    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls podName:e153e566-64cd-4297-9078-65b0e8600477 nodeName:}" failed. No retries permitted until 2026-07-27 21:16:30.494118794 +0000 UTC m=+719.158744012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls") pod "isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" (UID: "e153e566-64cd-4297-9078-65b0e8600477") : secret "isvc-sklearn-graph-raw-6148e-predictor-serving-cert" not found
Jul 27 21:16:29.994471 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.994453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:29.994525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:29.994453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:30.003323 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.003304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxss6\" (UniqueName: \"kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:30.094719 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.094649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.094719 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.094694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.094719 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.094721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q4k\" (UniqueName: \"kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.095004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.094736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.095004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.094764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.095129 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.095104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.095520 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.095501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.095520 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.095515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.097312 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.097294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.102609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.102588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q4k\" (UniqueName: \"kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k\") pod \"isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.229035 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.229006    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e43173-641a-4b63-8439-0cc93b7379e4" containerID="7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c" exitCode=2
Jul 27 21:16:30.229175 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.229088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerDied","Data":"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"}
Jul 27 21:16:30.231333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.231318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:30.368753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.368727    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:16:30.370602 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:16:30.370579    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3704c7_2459_440a_8cf8_d34758972f77.slice/crio-6b6e708fa3267475f44edef93f1660cab4aa9bc3cf0302c20cbe7bfaa9a1ddb9 WatchSource:0}: Error finding container 6b6e708fa3267475f44edef93f1660cab4aa9bc3cf0302c20cbe7bfaa9a1ddb9: Status 404 returned error can't find the container with id 6b6e708fa3267475f44edef93f1660cab4aa9bc3cf0302c20cbe7bfaa9a1ddb9
Jul 27 21:16:30.498543 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.498514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:30.500843 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.500824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") pod \"isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:30.761865 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.761772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:30.901024 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:30.900999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:16:30.903481 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:16:30.903456    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode153e566_64cd_4297_9078_65b0e8600477.slice/crio-4bf1a9eec6a74106185e8fefcc44f0ae89458559e111e08e610ecb7b316bd9fb WatchSource:0}: Error finding container 4bf1a9eec6a74106185e8fefcc44f0ae89458559e111e08e610ecb7b316bd9fb: Status 404 returned error can't find the container with id 4bf1a9eec6a74106185e8fefcc44f0ae89458559e111e08e610ecb7b316bd9fb
Jul 27 21:16:31.234899 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:31.234869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerStarted","Data":"ac4633cfc0c87339149402867b3af7e16fb8c61f1159b1eb13b08f10caf1bfac"}
Jul 27 21:16:31.235099 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:31.234929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerStarted","Data":"4bf1a9eec6a74106185e8fefcc44f0ae89458559e111e08e610ecb7b316bd9fb"}
Jul 27 21:16:31.236551 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:31.236529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerStarted","Data":"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"}
Jul 27 21:16:31.236651 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:31.236578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerStarted","Data":"6b6e708fa3267475f44edef93f1660cab4aa9bc3cf0302c20cbe7bfaa9a1ddb9"}
Jul 27 21:16:34.255022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:34.254995    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e43173-641a-4b63-8439-0cc93b7379e4" containerID="e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474" exitCode=0
Jul 27 21:16:34.255321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:34.255072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerDied","Data":"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"}
Jul 27 21:16:34.256431 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:34.256414    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c3704c7-2459-440a-8cf8-d34758972f77" containerID="be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e" exitCode=0
Jul 27 21:16:34.256490 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:34.256456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerDied","Data":"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"}
Jul 27 21:16:34.831398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:34.831352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:35.265482 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:35.265388    2580 generic.go:356] "Generic (PLEG): container finished" podID="e153e566-64cd-4297-9078-65b0e8600477" containerID="ac4633cfc0c87339149402867b3af7e16fb8c61f1159b1eb13b08f10caf1bfac" exitCode=0
Jul 27 21:16:35.265482 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:35.265455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerDied","Data":"ac4633cfc0c87339149402867b3af7e16fb8c61f1159b1eb13b08f10caf1bfac"}
Jul 27 21:16:35.836123 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:35.836071    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:16:35.836602 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:35.836541    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:37.280727 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.280681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerStarted","Data":"0fcb221a2dce25ff0d115fb7bb4cd9aa7c260031331cd003625887453c65cc8a"}
Jul 27 21:16:37.281199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.280770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerStarted","Data":"c6963909983a10e4603332f6dabed0563b3a8458e7415a47f954a5f30d6e41be"}
Jul 27 21:16:37.281199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.281048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:37.281338 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.281232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:37.282564 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.282537    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:16:37.300638 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:37.300589    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podStartSLOduration=4.453699345 podStartE2EDuration="8.300574407s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.846875062s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:37.298242643 +0000 UTC m=+725.962867897" watchObservedRunningTime="2026-07-27 21:16:37.300574407 +0000 UTC m=+725.965199645"
Jul 27 21:16:38.285524 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:38.285486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:16:39.831479 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:39.831425    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:39.831914 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:39.831582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:16:43.291143 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:43.291115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:16:43.291717 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:43.291687    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:16:44.831245 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:44.831201    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:45.836447 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:45.836391    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:16:45.836880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:45.836703    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:49.831840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:49.831795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:53.292080 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:53.292039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:16:54.371160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.371128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerStarted","Data":"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"}
Jul 27 21:16:54.371559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.371188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerStarted","Data":"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"}
Jul 27 21:16:54.371559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.371486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:54.371660 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.371623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:16:54.372789 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.372765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:16:54.393432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.393379    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podStartSLOduration=2.053554824 podStartE2EDuration="25.393363887s" totalImagesPullingTime="19.568829055s" totalInitContainerRuntime="3.770980008s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:54.391351606 +0000 UTC m=+743.055976868" watchObservedRunningTime="2026-07-27 21:16:54.393363887 +0000 UTC m=+743.057989194"
Jul 27 21:16:54.831621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:54.831585    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:55.376435 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:55.376400    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:16:55.836592 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:55.836555    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:16:55.836778 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:55.836690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:16:55.836844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:55.836786    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:55.836902 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:55.836886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:16:59.831083 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:59.831047    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:16:59.995241 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:16:59.995219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:17:00.172590 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location\") pod \"29e43173-641a-4b63-8439-0cc93b7379e4\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") "
Jul 27 21:17:00.172590 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config\") pod \"29e43173-641a-4b63-8439-0cc93b7379e4\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") "
Jul 27 21:17:00.172796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") pod \"29e43173-641a-4b63-8439-0cc93b7379e4\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") "
Jul 27 21:17:00.172796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpcbk\" (UniqueName: \"kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk\") pod \"29e43173-641a-4b63-8439-0cc93b7379e4\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") "
Jul 27 21:17:00.172796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert\") pod \"29e43173-641a-4b63-8439-0cc93b7379e4\" (UID: \"29e43173-641a-4b63-8439-0cc93b7379e4\") "
Jul 27 21:17:00.172939 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172823    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29e43173-641a-4b63-8439-0cc93b7379e4" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:17:00.173052 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.172926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29e43173-641a-4b63-8439-0cc93b7379e4" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:17:00.173189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.173096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29e43173-641a-4b63-8439-0cc93b7379e4" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:17:00.173189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.173124    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e43173-641a-4b63-8439-0cc93b7379e4-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.173189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.173145    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.175061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.175031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29e43173-641a-4b63-8439-0cc93b7379e4" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:17:00.175061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.175046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk" (OuterVolumeSpecName: "kube-api-access-tpcbk") pod "29e43173-641a-4b63-8439-0cc93b7379e4" (UID: "29e43173-641a-4b63-8439-0cc93b7379e4"). InnerVolumeSpecName "kube-api-access-tpcbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:17:00.273790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.273747    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e43173-641a-4b63-8439-0cc93b7379e4-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.273790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.273783    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcbk\" (UniqueName: \"kubernetes.io/projected/29e43173-641a-4b63-8439-0cc93b7379e4-kube-api-access-tpcbk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.273790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.273793    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e43173-641a-4b63-8439-0cc93b7379e4-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:17:00.380273 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.380234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:17:00.380844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.380817    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:17:00.400952 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.400930    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e43173-641a-4b63-8439-0cc93b7379e4" containerID="678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3" exitCode=0
Jul 27 21:17:00.401093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.400999    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"
Jul 27 21:17:00.401093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.400937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerDied","Data":"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"}
Jul 27 21:17:00.401093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.401066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp" event={"ID":"29e43173-641a-4b63-8439-0cc93b7379e4","Type":"ContainerDied","Data":"ec56542498ade1ae2ea49cd23cd85bbc24604ae3355ac37ecc4019af581ca266"}
Jul 27 21:17:00.401207 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.401100    2580 scope.go:117] "RemoveContainer" containerID="678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"
Jul 27 21:17:00.410218 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.410199    2580 scope.go:117] "RemoveContainer" containerID="7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"
Jul 27 21:17:00.417579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.417561    2580 scope.go:117] "RemoveContainer" containerID="e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"
Jul 27 21:17:00.421926 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.421908    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:17:00.424681 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.424638    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp"]
Jul 27 21:17:00.425466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.425449    2580 scope.go:117] "RemoveContainer" containerID="8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"
Jul 27 21:17:00.432857 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.432843    2580 scope.go:117] "RemoveContainer" containerID="678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"
Jul 27 21:17:00.433139 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:17:00.433118    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3\": container with ID starting with 678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3 not found: ID does not exist" containerID="678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"
Jul 27 21:17:00.433203 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433161    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3"} err="failed to get container status \"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3\": rpc error: code = NotFound desc = could not find container \"678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3\": container with ID starting with 678f112de26544c49424138f5e87ff8259e3834a8d617e12fd53d5c3736d7de3 not found: ID does not exist"
Jul 27 21:17:00.433203 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433192    2580 scope.go:117] "RemoveContainer" containerID="7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"
Jul 27 21:17:00.433448 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:17:00.433430    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c\": container with ID starting with 7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c not found: ID does not exist" containerID="7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"
Jul 27 21:17:00.433506 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433455    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c"} err="failed to get container status \"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c\": rpc error: code = NotFound desc = could not find container \"7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c\": container with ID starting with 7d4021834fde6040ffaa98c6a40c27b688cdc5c2478ebe1b118fe8561306286c not found: ID does not exist"
Jul 27 21:17:00.433506 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433471    2580 scope.go:117] "RemoveContainer" containerID="e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"
Jul 27 21:17:00.433688 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:17:00.433674    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474\": container with ID starting with e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474 not found: ID does not exist" containerID="e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"
Jul 27 21:17:00.433731 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474"} err="failed to get container status \"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474\": rpc error: code = NotFound desc = could not find container \"e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474\": container with ID starting with e26a5b413716978a2a5b92ca12e4074fa6ec6d7dd2ce0305ff82ca1f59451474 not found: ID does not exist"
Jul 27 21:17:00.433731 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433702    2580 scope.go:117] "RemoveContainer" containerID="8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"
Jul 27 21:17:00.433905 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:17:00.433892    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e\": container with ID starting with 8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e not found: ID does not exist" containerID="8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"
Jul 27 21:17:00.433936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:00.433908    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e"} err="failed to get container status \"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e\": rpc error: code = NotFound desc = could not find container \"8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e\": container with ID starting with 8c4127919194da307d1facd3917f9559bf51f603287239678dfb2cd09ebf223e not found: ID does not exist"
Jul 27 21:17:01.877898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:01.877869    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" path="/var/lib/kubelet/pods/29e43173-641a-4b63-8439-0cc93b7379e4/volumes"
Jul 27 21:17:03.291621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:03.291587    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:10.381466 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:10.381426    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:17:13.292333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:13.292297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:20.381605 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:20.381562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:17:23.291754 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:23.291705    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:30.381337 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:30.381300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:17:33.291915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:33.291876    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:40.380751 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:40.380714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:17:43.291646 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:43.291607    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:48.872853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:48.872815    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 21:17:50.381802 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:50.381777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:17:58.873981 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:17:58.873905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:18:20.151740 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.151709    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:18:20.152288 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.151958    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container" containerID="cri-o://c6963909983a10e4603332f6dabed0563b3a8458e7415a47f954a5f30d6e41be" gracePeriod=30
Jul 27 21:18:20.152288 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.151980    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kube-rbac-proxy" containerID="cri-o://0fcb221a2dce25ff0d115fb7bb4cd9aa7c260031331cd003625887453c65cc8a" gracePeriod=30
Jul 27 21:18:20.157824 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.157799    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:18:20.158775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158754    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent"
Jul 27 21:18:20.158872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158780    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent"
Jul 27 21:18:20.158872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158804    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="storage-initializer"
Jul 27 21:18:20.158872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="storage-initializer"
Jul 27 21:18:20.158872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container"
Jul 27 21:18:20.158872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158862    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container"
Jul 27 21:18:20.159059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy"
Jul 27 21:18:20.159059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158882    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy"
Jul 27 21:18:20.159059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kserve-container"
Jul 27 21:18:20.159059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.158993    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="agent"
Jul 27 21:18:20.159059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.159000    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e43173-641a-4b63-8439-0cc93b7379e4" containerName="kube-rbac-proxy"
Jul 27 21:18:20.161670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.161650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.164711 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.164690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-1559b-kube-rbac-proxy-sar-config\""
Jul 27 21:18:20.165198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.164690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-1559b-predictor-serving-cert\""
Jul 27 21:18:20.173406 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.173385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:18:20.239028 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.239003    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:18:20.240549 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.240525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.240668 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.240577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.240745 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.240670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k68d\" (UniqueName: \"kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.240745 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.240717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.240840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.240742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.241879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.241858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.244760 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.244672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-1559b-predictor-serving-cert\""
Jul 27 21:18:20.244760 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.244707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-1559b-kube-rbac-proxy-sar-config\""
Jul 27 21:18:20.254418 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.254297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:18:20.280908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.280882    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:18:20.281136 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.281112    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" containerID="cri-o://4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8" gracePeriod=30
Jul 27 21:18:20.281202 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.281155    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kube-rbac-proxy" containerID="cri-o://f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f" gracePeriod=30
Jul 27 21:18:20.341366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.341500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.341500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxrs\" (UniqueName: \"kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.341500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.341500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.341500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k68d\" (UniqueName: \"kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.341746 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.341746 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.341746 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.341746 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.341746 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:20.341710    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-1559b-predictor-serving-cert" not found
Jul 27 21:18:20.341942 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:20.341776    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls podName:11bc33ac-acc3-4819-b1b3-14858f3be5f0 nodeName:}" failed. No retries permitted until 2026-07-27 21:18:20.841755438 +0000 UTC m=+829.506380654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0") : secret "isvc-sklearn-graph-raw-hpa-1559b-predictor-serving-cert" not found
Jul 27 21:18:20.341942 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.341915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.342155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.342133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.342195 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.342133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.350275 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.350236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k68d\" (UniqueName: \"kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.377393 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.377357    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 27 21:18:20.381666 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.381645    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 21:18:20.442157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxrs\" (UniqueName: \"kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442419 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:20.442356    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-1559b-predictor-serving-cert" not found
Jul 27 21:18:20.442419 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:20.442416    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls podName:c409df00-43ec-4b68-80eb-0c897a2c18f0 nodeName:}" failed. No retries permitted until 2026-07-27 21:18:20.94239579 +0000 UTC m=+829.607021008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0") : secret "isvc-xgboost-graph-raw-hpa-1559b-predictor-serving-cert" not found
Jul 27 21:18:20.442501 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442773 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442809 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.442871 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.442855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.450357 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.450337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxrs\" (UniqueName: \"kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.795243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.795155    2580 generic.go:356] "Generic (PLEG): container finished" podID="e153e566-64cd-4297-9078-65b0e8600477" containerID="0fcb221a2dce25ff0d115fb7bb4cd9aa7c260031331cd003625887453c65cc8a" exitCode=2
Jul 27 21:18:20.795415 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.795234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerDied","Data":"0fcb221a2dce25ff0d115fb7bb4cd9aa7c260031331cd003625887453c65cc8a"}
Jul 27 21:18:20.797327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.797306    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c3704c7-2459-440a-8cf8-d34758972f77" containerID="f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f" exitCode=2
Jul 27 21:18:20.797441 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.797360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerDied","Data":"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"}
Jul 27 21:18:20.844868 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.844848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.847318 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.847300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:20.945839 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.945807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:20.948227 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:20.948209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:21.083985 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.083940    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:21.154778 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.154748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:21.244164 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.244141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:18:21.246378 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:18:21.246346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bc33ac_acc3_4819_b1b3_14858f3be5f0.slice/crio-93fde493536a352b1f9c904e234f52129233a27ba85bcc3d32671cd26413fa90 WatchSource:0}: Error finding container 93fde493536a352b1f9c904e234f52129233a27ba85bcc3d32671cd26413fa90: Status 404 returned error can't find the container with id 93fde493536a352b1f9c904e234f52129233a27ba85bcc3d32671cd26413fa90
Jul 27 21:18:21.310937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.310805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:18:21.313652 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:18:21.313616    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc409df00_43ec_4b68_80eb_0c897a2c18f0.slice/crio-12d51e0860f2e9e999f296f287ccefcc2e598980bfefaed0b05adc7837824114 WatchSource:0}: Error finding container 12d51e0860f2e9e999f296f287ccefcc2e598980bfefaed0b05adc7837824114: Status 404 returned error can't find the container with id 12d51e0860f2e9e999f296f287ccefcc2e598980bfefaed0b05adc7837824114
Jul 27 21:18:21.804790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.804756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerStarted","Data":"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"}
Jul 27 21:18:21.804968 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.804828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerStarted","Data":"12d51e0860f2e9e999f296f287ccefcc2e598980bfefaed0b05adc7837824114"}
Jul 27 21:18:21.806408 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.806385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerStarted","Data":"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"}
Jul 27 21:18:21.806514 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:21.806432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerStarted","Data":"93fde493536a352b1f9c904e234f52129233a27ba85bcc3d32671cd26413fa90"}
Jul 27 21:18:23.286529 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.286490    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 21:18:23.735271 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.735232    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:18:23.770403 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls\") pod \"8c3704c7-2459-440a-8cf8-d34758972f77\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") "
Jul 27 21:18:23.770539 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location\") pod \"8c3704c7-2459-440a-8cf8-d34758972f77\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") "
Jul 27 21:18:23.770539 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6q4k\" (UniqueName: \"kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k\") pod \"8c3704c7-2459-440a-8cf8-d34758972f77\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") "
Jul 27 21:18:23.770539 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert\") pod \"8c3704c7-2459-440a-8cf8-d34758972f77\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") "
Jul 27 21:18:23.770682 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config\") pod \"8c3704c7-2459-440a-8cf8-d34758972f77\" (UID: \"8c3704c7-2459-440a-8cf8-d34758972f77\") "
Jul 27 21:18:23.770742 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c3704c7-2459-440a-8cf8-d34758972f77" (UID: "8c3704c7-2459-440a-8cf8-d34758972f77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:18:23.770868 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770850    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c3704c7-2459-440a-8cf8-d34758972f77-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:23.770932 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c3704c7-2459-440a-8cf8-d34758972f77" (UID: "8c3704c7-2459-440a-8cf8-d34758972f77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:23.770977 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.770947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c3704c7-2459-440a-8cf8-d34758972f77" (UID: "8c3704c7-2459-440a-8cf8-d34758972f77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:23.772595 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.772574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k" (OuterVolumeSpecName: "kube-api-access-s6q4k") pod "8c3704c7-2459-440a-8cf8-d34758972f77" (UID: "8c3704c7-2459-440a-8cf8-d34758972f77"). InnerVolumeSpecName "kube-api-access-s6q4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:18:23.772963 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.772950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c3704c7-2459-440a-8cf8-d34758972f77" (UID: "8c3704c7-2459-440a-8cf8-d34758972f77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:18:23.817006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.816941    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c3704c7-2459-440a-8cf8-d34758972f77" containerID="4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8" exitCode=0
Jul 27 21:18:23.817110 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.817004    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"
Jul 27 21:18:23.817110 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.817031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerDied","Data":"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"}
Jul 27 21:18:23.817110 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.817102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s" event={"ID":"8c3704c7-2459-440a-8cf8-d34758972f77","Type":"ContainerDied","Data":"6b6e708fa3267475f44edef93f1660cab4aa9bc3cf0302c20cbe7bfaa9a1ddb9"}
Jul 27 21:18:23.817218 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.817129    2580 scope.go:117] "RemoveContainer" containerID="f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"
Jul 27 21:18:23.834336 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.834314    2580 scope.go:117] "RemoveContainer" containerID="4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"
Jul 27 21:18:23.842022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.842005    2580 scope.go:117] "RemoveContainer" containerID="be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"
Jul 27 21:18:23.847967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.847946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:18:23.850686 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.850668    2580 scope.go:117] "RemoveContainer" containerID="f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"
Jul 27 21:18:23.850931 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:23.850913    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f\": container with ID starting with f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f not found: ID does not exist" containerID="f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"
Jul 27 21:18:23.850975 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.850939    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f"} err="failed to get container status \"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f\": rpc error: code = NotFound desc = could not find container \"f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f\": container with ID starting with f1f23f3750fa330bc8274517a2860370b10562110291c23c24a7b2792ccd5a6f not found: ID does not exist"
Jul 27 21:18:23.850975 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.850961    2580 scope.go:117] "RemoveContainer" containerID="4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"
Jul 27 21:18:23.851182 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:23.851165    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8\": container with ID starting with 4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8 not found: ID does not exist" containerID="4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"
Jul 27 21:18:23.851222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.851188    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8"} err="failed to get container status \"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8\": rpc error: code = NotFound desc = could not find container \"4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8\": container with ID starting with 4f46300b6af0bea15d93af8943811fe2016deb2b832f083819f9b96782ac28f8 not found: ID does not exist"
Jul 27 21:18:23.851222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.851204    2580 scope.go:117] "RemoveContainer" containerID="be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"
Jul 27 21:18:23.851445 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:18:23.851429    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e\": container with ID starting with be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e not found: ID does not exist" containerID="be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"
Jul 27 21:18:23.851490 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.851445    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s"]
Jul 27 21:18:23.851490 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.851450    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e"} err="failed to get container status \"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e\": rpc error: code = NotFound desc = could not find container \"be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e\": container with ID starting with be27726fe2a6eafc58197f0c7f5f7d97dadf0d9bf9032f3999695a490010e99e not found: ID does not exist"
Jul 27 21:18:23.871866 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.871843    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:23.871866 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.871863    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c3704c7-2459-440a-8cf8-d34758972f77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:23.871981 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.871873    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c3704c7-2459-440a-8cf8-d34758972f77-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:23.871981 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.871882    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q4k\" (UniqueName: \"kubernetes.io/projected/8c3704c7-2459-440a-8cf8-d34758972f77-kube-api-access-s6q4k\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:23.878023 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:23.877997    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" path="/var/lib/kubelet/pods/8c3704c7-2459-440a-8cf8-d34758972f77/volumes"
Jul 27 21:18:24.826880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:24.826851    2580 generic.go:356] "Generic (PLEG): container finished" podID="e153e566-64cd-4297-9078-65b0e8600477" containerID="c6963909983a10e4603332f6dabed0563b3a8458e7415a47f954a5f30d6e41be" exitCode=0
Jul 27 21:18:24.827294 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:24.826900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerDied","Data":"c6963909983a10e4603332f6dabed0563b3a8458e7415a47f954a5f30d6e41be"}
Jul 27 21:18:24.946961 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:24.946906    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:18:25.082041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxss6\" (UniqueName: \"kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6\") pod \"e153e566-64cd-4297-9078-65b0e8600477\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") "
Jul 27 21:18:25.082184 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location\") pod \"e153e566-64cd-4297-9078-65b0e8600477\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") "
Jul 27 21:18:25.082184 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config\") pod \"e153e566-64cd-4297-9078-65b0e8600477\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") "
Jul 27 21:18:25.082184 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert\") pod \"e153e566-64cd-4297-9078-65b0e8600477\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") "
Jul 27 21:18:25.082184 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") pod \"e153e566-64cd-4297-9078-65b0e8600477\" (UID: \"e153e566-64cd-4297-9078-65b0e8600477\") "
Jul 27 21:18:25.082486 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082458    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e153e566-64cd-4297-9078-65b0e8600477" (UID: "e153e566-64cd-4297-9078-65b0e8600477"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:18:25.082557 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e153e566-64cd-4297-9078-65b0e8600477" (UID: "e153e566-64cd-4297-9078-65b0e8600477"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:25.082557 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.082513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e153e566-64cd-4297-9078-65b0e8600477" (UID: "e153e566-64cd-4297-9078-65b0e8600477"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:25.084476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.084454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e153e566-64cd-4297-9078-65b0e8600477" (UID: "e153e566-64cd-4297-9078-65b0e8600477"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:18:25.084678 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.084651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6" (OuterVolumeSpecName: "kube-api-access-mxss6") pod "e153e566-64cd-4297-9078-65b0e8600477" (UID: "e153e566-64cd-4297-9078-65b0e8600477"). InnerVolumeSpecName "kube-api-access-mxss6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:18:25.182945 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.182905    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:25.182945 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.182938    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e153e566-64cd-4297-9078-65b0e8600477-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:25.182945 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.182948    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e153e566-64cd-4297-9078-65b0e8600477-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:25.183176 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.182956    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxss6\" (UniqueName: \"kubernetes.io/projected/e153e566-64cd-4297-9078-65b0e8600477-kube-api-access-mxss6\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:25.183176 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.182965    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e153e566-64cd-4297-9078-65b0e8600477-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:18:25.833998 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.833968    2580 generic.go:356] "Generic (PLEG): container finished" podID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerID="a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df" exitCode=0
Jul 27 21:18:25.834517 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.834053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerDied","Data":"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"}
Jul 27 21:18:25.836085 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.836060    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"
Jul 27 21:18:25.836205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.836081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2" event={"ID":"e153e566-64cd-4297-9078-65b0e8600477","Type":"ContainerDied","Data":"4bf1a9eec6a74106185e8fefcc44f0ae89458559e111e08e610ecb7b316bd9fb"}
Jul 27 21:18:25.836205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.836147    2580 scope.go:117] "RemoveContainer" containerID="0fcb221a2dce25ff0d115fb7bb4cd9aa7c260031331cd003625887453c65cc8a"
Jul 27 21:18:25.837895 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.837874    2580 generic.go:356] "Generic (PLEG): container finished" podID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerID="799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8" exitCode=0
Jul 27 21:18:25.837992 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.837931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerDied","Data":"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"}
Jul 27 21:18:25.847413 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.847390    2580 scope.go:117] "RemoveContainer" containerID="c6963909983a10e4603332f6dabed0563b3a8458e7415a47f954a5f30d6e41be"
Jul 27 21:18:25.856033 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.856003    2580 scope.go:117] "RemoveContainer" containerID="ac4633cfc0c87339149402867b3af7e16fb8c61f1159b1eb13b08f10caf1bfac"
Jul 27 21:18:25.867741 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.867718    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:18:25.871088 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.871068    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2"]
Jul 27 21:18:25.877887 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:25.877866    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e153e566-64cd-4297-9078-65b0e8600477" path="/var/lib/kubelet/pods/e153e566-64cd-4297-9078-65b0e8600477/volumes"
Jul 27 21:18:26.844794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.844760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerStarted","Data":"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"}
Jul 27 21:18:26.845284 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.844821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerStarted","Data":"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"}
Jul 27 21:18:26.845284 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.845033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:26.847818 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.847798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerStarted","Data":"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"}
Jul 27 21:18:26.847921 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.847836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerStarted","Data":"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"}
Jul 27 21:18:26.848150 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.848136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:26.848285 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.848251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:26.849636 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.849612    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:18:26.864832 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.864631    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podStartSLOduration=3.241175395 podStartE2EDuration="6.864616821s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.623441426s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:26.862435221 +0000 UTC m=+835.527060476" watchObservedRunningTime="2026-07-27 21:18:26.864616821 +0000 UTC m=+835.529242059"
Jul 27 21:18:26.881705 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:26.881663    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podStartSLOduration=3.325986463 podStartE2EDuration="6.881650933s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.55566447s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:26.881068135 +0000 UTC m=+835.545693374" watchObservedRunningTime="2026-07-27 21:18:26.881650933 +0000 UTC m=+835.546276170"
Jul 27 21:18:27.852368 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:27.852326    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:18:27.852800 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:27.852436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:27.853505 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:27.853482    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:18:28.856690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:28.856642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:18:32.856293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:32.856248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:18:32.856852 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:32.856830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:18:33.862528 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:33.862501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:18:33.863108 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:33.863082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:18:42.857582 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:42.857548    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:18:43.862979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:43.862943    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:18:52.856886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:52.856841    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:18:53.863734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:18:53.863691    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:19:02.857714 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:02.857672    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:19:03.863666 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:03.863626    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:19:12.857389 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:12.857350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:19:13.863185 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:13.863141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:19:22.857243 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:22.857156    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:19:23.863677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:23.863653    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:19:32.857236 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:32.857191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:19:42.858211 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:19:42.858182    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:20:10.439988 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.439948    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:20:10.441172 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441148    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="storage-initializer"
Jul 27 21:20:10.441331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441316    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="storage-initializer"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441339    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kube-rbac-proxy"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441348    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kube-rbac-proxy"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441367    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441377    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441398    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container"
Jul 27 21:20:10.441414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441408    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441425    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="storage-initializer"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441433    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="storage-initializer"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kube-rbac-proxy"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441453    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kube-rbac-proxy"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441562    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kserve-container"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441575    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kserve-container"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441589    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3704c7-2459-440a-8cf8-d34758972f77" containerName="kube-rbac-proxy"
Jul 27 21:20:10.441706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.441600    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e153e566-64cd-4297-9078-65b0e8600477" containerName="kube-rbac-proxy"
Jul 27 21:20:10.444915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.444890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.447909 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.447844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4bc18-predictor-serving-cert\""
Jul 27 21:20:10.448414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.448393    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4bc18-kube-rbac-proxy-sar-config\""
Jul 27 21:20:10.454507 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.454486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:20:10.480546 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.480516    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:20:10.480813 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.480784    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" containerID="cri-o://a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e" gracePeriod=30
Jul 27 21:20:10.481039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.480856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kube-rbac-proxy" containerID="cri-o://6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7" gracePeriod=30
Jul 27 21:20:10.485013 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.484990    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:20:10.485231 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.485197    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" containerID="cri-o://29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e" gracePeriod=30
Jul 27 21:20:10.485324 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.485296    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kube-rbac-proxy" containerID="cri-o://19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e" gracePeriod=30
Jul 27 21:20:10.532189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.532157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptq26\" (UniqueName: \"kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.532315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.532205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.532315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.532283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.632630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.632598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.632738 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.632642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.632799 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:10.632743    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-serving-cert: secret "message-dumper-raw-4bc18-predictor-serving-cert" not found
Jul 27 21:20:10.632799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.632753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptq26\" (UniqueName: \"kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.632895 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:10.632804    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls podName:78a0e727-f04a-4aad-b1c2-c9e018041cda nodeName:}" failed. No retries permitted until 2026-07-27 21:20:11.132783196 +0000 UTC m=+939.797408414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls") pod "message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" (UID: "78a0e727-f04a-4aad-b1c2-c9e018041cda") : secret "message-dumper-raw-4bc18-predictor-serving-cert" not found
Jul 27 21:20:10.633295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.633247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:10.641750 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:10.641727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptq26\" (UniqueName: \"kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:11.138153 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.138118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:11.140675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.140650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") pod \"message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") " pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:11.328907 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.328877    2580 generic.go:356] "Generic (PLEG): container finished" podID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerID="6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7" exitCode=2
Jul 27 21:20:11.329061 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.328948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerDied","Data":"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"}
Jul 27 21:20:11.330936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.330917    2580 generic.go:356] "Generic (PLEG): container finished" podID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerID="19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e" exitCode=2
Jul 27 21:20:11.331044 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.330987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerDied","Data":"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"}
Jul 27 21:20:11.361357 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.361335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:11.491284 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.491246    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:20:11.492955 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:20:11.492931    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a0e727_f04a_4aad_b1c2_c9e018041cda.slice/crio-23768306d51991f0122532483615de2d5ea3882c160e65f2ed54497bc21fa131 WatchSource:0}: Error finding container 23768306d51991f0122532483615de2d5ea3882c160e65f2ed54497bc21fa131: Status 404 returned error can't find the container with id 23768306d51991f0122532483615de2d5ea3882c160e65f2ed54497bc21fa131
Jul 27 21:20:11.495204 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:11.495190    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:20:12.337622 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:12.337585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerStarted","Data":"23768306d51991f0122532483615de2d5ea3882c160e65f2ed54497bc21fa131"}
Jul 27 21:20:12.853389 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:12.853347    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 27 21:20:12.857656 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:12.857626    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:20:13.343639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.343602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerStarted","Data":"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"}
Jul 27 21:20:13.343844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.343672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerStarted","Data":"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"}
Jul 27 21:20:13.343844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.343706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:13.343960 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.343849    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:13.345817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.345799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:13.360978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.360932    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" podStartSLOduration=2.211287123 podStartE2EDuration="3.360917978s" totalImagesPullingTime="1.149630855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:13.36048528 +0000 UTC m=+942.025110535" watchObservedRunningTime="2026-07-27 21:20:13.360917978 +0000 UTC m=+942.025543216"
Jul 27 21:20:13.856828 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.856787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 27 21:20:13.863939 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:13.863903    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:20:14.145336 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.145307    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:20:14.164682 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.164653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnxrs\" (UniqueName: \"kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs\") pod \"c409df00-43ec-4b68-80eb-0c897a2c18f0\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") "
Jul 27 21:20:14.164854 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.164800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location\") pod \"c409df00-43ec-4b68-80eb-0c897a2c18f0\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") "
Jul 27 21:20:14.164854 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.164835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") pod \"c409df00-43ec-4b68-80eb-0c897a2c18f0\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") "
Jul 27 21:20:14.165000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.164871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config\") pod \"c409df00-43ec-4b68-80eb-0c897a2c18f0\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") "
Jul 27 21:20:14.165000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.164906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert\") pod \"c409df00-43ec-4b68-80eb-0c897a2c18f0\" (UID: \"c409df00-43ec-4b68-80eb-0c897a2c18f0\") "
Jul 27 21:20:14.165493 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.165341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c409df00-43ec-4b68-80eb-0c897a2c18f0" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:14.165651 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.165544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c409df00-43ec-4b68-80eb-0c897a2c18f0" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:14.165708 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.165679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c409df00-43ec-4b68-80eb-0c897a2c18f0" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:14.168382 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.168360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs" (OuterVolumeSpecName: "kube-api-access-qnxrs") pod "c409df00-43ec-4b68-80eb-0c897a2c18f0" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0"). InnerVolumeSpecName "kube-api-access-qnxrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:14.168957 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.168930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c409df00-43ec-4b68-80eb-0c897a2c18f0" (UID: "c409df00-43ec-4b68-80eb-0c897a2c18f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:14.266130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.266098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxrs\" (UniqueName: \"kubernetes.io/projected/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-api-access-qnxrs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:14.266130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.266124    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c409df00-43ec-4b68-80eb-0c897a2c18f0-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:14.266130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.266134    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c409df00-43ec-4b68-80eb-0c897a2c18f0-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:14.266370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.266143    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:14.266370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.266153    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c409df00-43ec-4b68-80eb-0c897a2c18f0-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:14.351711 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.351675    2580 generic.go:356] "Generic (PLEG): container finished" podID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerID="29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e" exitCode=0
Jul 27 21:20:14.351861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.351739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"
Jul 27 21:20:14.351861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.351740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerDied","Data":"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"}
Jul 27 21:20:14.351861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.351802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x" event={"ID":"c409df00-43ec-4b68-80eb-0c897a2c18f0","Type":"ContainerDied","Data":"12d51e0860f2e9e999f296f287ccefcc2e598980bfefaed0b05adc7837824114"}
Jul 27 21:20:14.351861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.351829    2580 scope.go:117] "RemoveContainer" containerID="19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"
Jul 27 21:20:14.361300 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.361282    2580 scope.go:117] "RemoveContainer" containerID="29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"
Jul 27 21:20:14.369098 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.369082    2580 scope.go:117] "RemoveContainer" containerID="a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"
Jul 27 21:20:14.373849 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.373830    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:20:14.378330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378315    2580 scope.go:117] "RemoveContainer" containerID="19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"
Jul 27 21:20:14.378618 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:14.378602    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e\": container with ID starting with 19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e not found: ID does not exist" containerID="19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"
Jul 27 21:20:14.378690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378626    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e"} err="failed to get container status \"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e\": rpc error: code = NotFound desc = could not find container \"19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e\": container with ID starting with 19124aad1037c6d3d1c6b1b7247274fdd60573cfda76fe56c5a26c2218fef66e not found: ID does not exist"
Jul 27 21:20:14.378690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378642    2580 scope.go:117] "RemoveContainer" containerID="29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"
Jul 27 21:20:14.378879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378857    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x"]
Jul 27 21:20:14.378940 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:14.378904    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e\": container with ID starting with 29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e not found: ID does not exist" containerID="29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"
Jul 27 21:20:14.378940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378928    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e"} err="failed to get container status \"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e\": rpc error: code = NotFound desc = could not find container \"29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e\": container with ID starting with 29ad78657b1d795087cff3c7f4ce165545e4118a199fad639b57ce3030615e9e not found: ID does not exist"
Jul 27 21:20:14.379030 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.378948    2580 scope.go:117] "RemoveContainer" containerID="a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"
Jul 27 21:20:14.379224 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:14.379201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df\": container with ID starting with a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df not found: ID does not exist" containerID="a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"
Jul 27 21:20:14.379325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:14.379232    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df"} err="failed to get container status \"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df\": rpc error: code = NotFound desc = could not find container \"a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df\": container with ID starting with a881650257a60d9112ab97cabe80d40dffecd1c7c7b9b98cbbb0d2d21ef775df not found: ID does not exist"
Jul 27 21:20:15.140020 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.139997    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:20:15.174182 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location\") pod \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") "
Jul 27 21:20:15.174325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert\") pod \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") "
Jul 27 21:20:15.174325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") pod \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") "
Jul 27 21:20:15.174325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174322    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config\") pod \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") "
Jul 27 21:20:15.174504 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k68d\" (UniqueName: \"kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d\") pod \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\" (UID: \"11bc33ac-acc3-4819-b1b3-14858f3be5f0\") "
Jul 27 21:20:15.174565 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174528    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11bc33ac-acc3-4819-b1b3-14858f3be5f0" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:15.174691 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11bc33ac-acc3-4819-b1b3-14858f3be5f0" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:15.174767 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174673    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:15.174767 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.174718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11bc33ac-acc3-4819-b1b3-14858f3be5f0" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:15.176571 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.176545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11bc33ac-acc3-4819-b1b3-14858f3be5f0" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:15.176660 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.176610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d" (OuterVolumeSpecName: "kube-api-access-2k68d") pod "11bc33ac-acc3-4819-b1b3-14858f3be5f0" (UID: "11bc33ac-acc3-4819-b1b3-14858f3be5f0"). InnerVolumeSpecName "kube-api-access-2k68d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:15.275978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.275891    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11bc33ac-acc3-4819-b1b3-14858f3be5f0-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:15.275978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.275929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:15.275978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.275938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k68d\" (UniqueName: \"kubernetes.io/projected/11bc33ac-acc3-4819-b1b3-14858f3be5f0-kube-api-access-2k68d\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:15.275978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.275948    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11bc33ac-acc3-4819-b1b3-14858f3be5f0-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:20:15.358085 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.358056    2580 generic.go:356] "Generic (PLEG): container finished" podID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerID="a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e" exitCode=0
Jul 27 21:20:15.358230 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.358111    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"
Jul 27 21:20:15.358230 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.358143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerDied","Data":"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"}
Jul 27 21:20:15.358230 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.358220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc" event={"ID":"11bc33ac-acc3-4819-b1b3-14858f3be5f0","Type":"ContainerDied","Data":"93fde493536a352b1f9c904e234f52129233a27ba85bcc3d32671cd26413fa90"}
Jul 27 21:20:15.358394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.358279    2580 scope.go:117] "RemoveContainer" containerID="6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"
Jul 27 21:20:15.367058 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.367040    2580 scope.go:117] "RemoveContainer" containerID="a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"
Jul 27 21:20:15.374456 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.374443    2580 scope.go:117] "RemoveContainer" containerID="799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"
Jul 27 21:20:15.380827 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.380808    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:20:15.382576 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.382495    2580 scope.go:117] "RemoveContainer" containerID="6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"
Jul 27 21:20:15.382974 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:15.382940    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7\": container with ID starting with 6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7 not found: ID does not exist" containerID="6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"
Jul 27 21:20:15.382974 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.382981    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7"} err="failed to get container status \"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7\": rpc error: code = NotFound desc = could not find container \"6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7\": container with ID starting with 6107f3076283fd217b90caf6ca56cdc47ffd551f2e4472c876279bccfd7dd6e7 not found: ID does not exist"
Jul 27 21:20:15.383151 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.383005    2580 scope.go:117] "RemoveContainer" containerID="a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"
Jul 27 21:20:15.383605 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:15.383577    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e\": container with ID starting with a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e not found: ID does not exist" containerID="a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"
Jul 27 21:20:15.383677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.383610    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e"} err="failed to get container status \"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e\": rpc error: code = NotFound desc = could not find container \"a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e\": container with ID starting with a893b9136fb581d138e486d33225c3cd8cae5c2b2db30fcb286a949f2bd63f0e not found: ID does not exist"
Jul 27 21:20:15.383677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.383650    2580 scope.go:117] "RemoveContainer" containerID="799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"
Jul 27 21:20:15.383900 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:20:15.383875    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8\": container with ID starting with 799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8 not found: ID does not exist" containerID="799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"
Jul 27 21:20:15.383957 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.383904    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8"} err="failed to get container status \"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8\": rpc error: code = NotFound desc = could not find container \"799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8\": container with ID starting with 799b22cfa3be73545329731af33d7ceee4543b22357218523235efa69cc063e8 not found: ID does not exist"
Jul 27 21:20:15.385032 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.385009    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc"]
Jul 27 21:20:15.882822 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.879957    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" path="/var/lib/kubelet/pods/11bc33ac-acc3-4819-b1b3-14858f3be5f0/volumes"
Jul 27 21:20:15.882822 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:15.880693    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" path="/var/lib/kubelet/pods/c409df00-43ec-4b68-80eb-0c897a2c18f0/volumes"
Jul 27 21:20:20.363882 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:20.363856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:20:30.444331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444296    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:20:30.444677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444664    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.444677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444676    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444687    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444692    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444702    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444707    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="storage-initializer"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444720    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="storage-initializer"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="storage-initializer"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444737    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="storage-initializer"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444745    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.444753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444750    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.445039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444802    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kserve-container"
Jul 27 21:20:30.445039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444814    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.445039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444821    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bc33ac-acc3-4819-b1b3-14858f3be5f0" containerName="kserve-container"
Jul 27 21:20:30.445039 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.444828    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c409df00-43ec-4b68-80eb-0c897a2c18f0" containerName="kube-rbac-proxy"
Jul 27 21:20:30.450237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.450219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.452916 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.452894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:20:30.453036 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.452992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4bc18-kube-rbac-proxy-sar-config\""
Jul 27 21:20:30.453220 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.453207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4bc18-predictor-serving-cert\""
Jul 27 21:20:30.453600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.453586    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:20:30.458342 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.458325    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:20:30.498144 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.498119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.498251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.498153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvpc\" (UniqueName: \"kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.498251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.498196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.498251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.498211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.498251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.498237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.599657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.599629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.599826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.599678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvpc\" (UniqueName: \"kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.599826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.599758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.599826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.599782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.599826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.599821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.600338 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.600310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.600576 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.600552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.600714 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.600670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.602560 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.602541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.608378 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.608348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvpc\" (UniqueName: \"kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc\") pod \"isvc-logger-raw-4bc18-predictor-595655c87f-p2psf\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.761500 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.761406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:30.895850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:30.895820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:20:30.898937 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:20:30.898912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7613dc26_bb6d_437b_85b1_a503350cbbdd.slice/crio-70604c2fccc1ba28f505935443f09f3b90479da6c596e978fd2cd519ed57d0d3 WatchSource:0}: Error finding container 70604c2fccc1ba28f505935443f09f3b90479da6c596e978fd2cd519ed57d0d3: Status 404 returned error can't find the container with id 70604c2fccc1ba28f505935443f09f3b90479da6c596e978fd2cd519ed57d0d3
Jul 27 21:20:31.440078 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:31.440045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerStarted","Data":"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"}
Jul 27 21:20:31.440270 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:31.440114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerStarted","Data":"70604c2fccc1ba28f505935443f09f3b90479da6c596e978fd2cd519ed57d0d3"}
Jul 27 21:20:35.459757 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:35.459720    2580 generic.go:356] "Generic (PLEG): container finished" podID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerID="54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5" exitCode=0
Jul 27 21:20:35.460130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:35.459792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerDied","Data":"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"}
Jul 27 21:20:36.466883 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.466842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerStarted","Data":"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"}
Jul 27 21:20:36.467246 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.466908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerStarted","Data":"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"}
Jul 27 21:20:36.467246 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.466931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerStarted","Data":"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"}
Jul 27 21:20:36.467374 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.467286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:36.467374 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.467357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:36.468663 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.468638    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:20:36.490707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:36.490663    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podStartSLOduration=2.792037565 podStartE2EDuration="6.490650639s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698613074s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:36.488052554 +0000 UTC m=+965.152677806" watchObservedRunningTime="2026-07-27 21:20:36.490650639 +0000 UTC m=+965.155275923"
Jul 27 21:20:37.471319 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:37.471285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:37.471684 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:37.471437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:20:37.472611 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:37.472584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:38.475579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:38.475537    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:20:38.475960 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:38.475936    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:43.479639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:43.479608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:20:43.480297 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:43.480240    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:20:43.480567 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:43.480546    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:53.480249 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:53.480156    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:20:53.480670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:20:53.480624    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:03.480635 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:03.480594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:21:03.481093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:03.481072    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:13.480443 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:13.480402    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:21:13.480928 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:13.480905    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:23.480160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:23.480122    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:21:23.480658 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:23.480485    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:33.480895 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:33.480855    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:21:33.481341 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:33.481198    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:43.480497 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:43.480457    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:21:43.480931 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:43.480910    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:53.481077 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:53.481051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:21:53.481510 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:21:53.481159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:05.462471 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.462445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn_78a0e727-f04a-4aad-b1c2-c9e018041cda/kserve-container/0.log"
Jul 27 21:22:05.681701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.681653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:22:05.682080 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.682024    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" containerID="cri-o://cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9" gracePeriod=30
Jul 27 21:22:05.682291 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.682037    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" containerID="cri-o://239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc" gracePeriod=30
Jul 27 21:22:05.682519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.682200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" containerID="cri-o://1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344" gracePeriod=30
Jul 27 21:22:05.755329 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.755249    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:22:05.759006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.758990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.761898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.761877    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dcfcf-kube-rbac-proxy-sar-config\""
Jul 27 21:22:05.761998 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.761902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dcfcf-predictor-serving-cert\""
Jul 27 21:22:05.771850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.771825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:22:05.792917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.792287    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:22:05.792917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.792563    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kserve-container" containerID="cri-o://63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114" gracePeriod=30
Jul 27 21:22:05.792917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.792734    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kube-rbac-proxy" containerID="cri-o://f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13" gracePeriod=30
Jul 27 21:22:05.876491 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.876460    2580 generic.go:356] "Generic (PLEG): container finished" podID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerID="1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344" exitCode=2
Jul 27 21:22:05.877359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.877339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerDied","Data":"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"}
Jul 27 21:22:05.883852 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.883834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.883918 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.883867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.883918 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.883887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfj7\" (UniqueName: \"kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.883918 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.883906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.884043 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.883988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.984786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.984888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.984925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:05.984951    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-serving-cert: secret "isvc-sklearn-scale-raw-dcfcf-predictor-serving-cert" not found
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.984953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfj7\" (UniqueName: \"kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:05.985026    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls podName:a8b61230-766e-42bb-af1d-b1a7b0d8bab4 nodeName:}" failed. No retries permitted until 2026-07-27 21:22:06.484997411 +0000 UTC m=+1055.149622629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls") pod "isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4") : secret "isvc-sklearn-scale-raw-dcfcf-predictor-serving-cert" not found
Jul 27 21:22:05.985272 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.985079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985653 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.985331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.985782 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.985759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.986016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.985999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:05.993737 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:05.993710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfj7\" (UniqueName: \"kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:06.040090 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.040066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:22:06.187237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.187206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptq26\" (UniqueName: \"kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26\") pod \"78a0e727-f04a-4aad-b1c2-c9e018041cda\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") "
Jul 27 21:22:06.187420 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.187264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") pod \"78a0e727-f04a-4aad-b1c2-c9e018041cda\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") "
Jul 27 21:22:06.187420 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.187331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config\") pod \"78a0e727-f04a-4aad-b1c2-c9e018041cda\" (UID: \"78a0e727-f04a-4aad-b1c2-c9e018041cda\") "
Jul 27 21:22:06.187706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.187681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78a0e727-f04a-4aad-b1c2-c9e018041cda" (UID: "78a0e727-f04a-4aad-b1c2-c9e018041cda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:06.189369 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.189351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78a0e727-f04a-4aad-b1c2-c9e018041cda" (UID: "78a0e727-f04a-4aad-b1c2-c9e018041cda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:06.189429 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.189373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26" (OuterVolumeSpecName: "kube-api-access-ptq26") pod "78a0e727-f04a-4aad-b1c2-c9e018041cda" (UID: "78a0e727-f04a-4aad-b1c2-c9e018041cda"). InnerVolumeSpecName "kube-api-access-ptq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:06.288609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.288547    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:06.288609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.288571    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptq26\" (UniqueName: \"kubernetes.io/projected/78a0e727-f04a-4aad-b1c2-c9e018041cda-kube-api-access-ptq26\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:06.288609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.288581    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a0e727-f04a-4aad-b1c2-c9e018041cda-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:06.490382 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.490352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:06.492781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.492763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") pod \"isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:06.669987 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.669958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:06.801976 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.801953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:22:06.882212 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882182    2580 generic.go:356] "Generic (PLEG): container finished" podID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerID="f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13" exitCode=2
Jul 27 21:22:06.882212 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882210    2580 generic.go:356] "Generic (PLEG): container finished" podID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerID="63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114" exitCode=2
Jul 27 21:22:06.882437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882237    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"
Jul 27 21:22:06.882437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerDied","Data":"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"}
Jul 27 21:22:06.882437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerDied","Data":"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"}
Jul 27 21:22:06.882437 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn" event={"ID":"78a0e727-f04a-4aad-b1c2-c9e018041cda","Type":"ContainerDied","Data":"23768306d51991f0122532483615de2d5ea3882c160e65f2ed54497bc21fa131"}
Jul 27 21:22:06.882589 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.882440    2580 scope.go:117] "RemoveContainer" containerID="f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"
Jul 27 21:22:06.884121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.884099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerStarted","Data":"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"}
Jul 27 21:22:06.884235 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.884147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerStarted","Data":"ba169d0e134b41ed23459dadccbae608e9534580b10b58261c4a9a6ba23a8701"}
Jul 27 21:22:06.891347 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.891332    2580 scope.go:117] "RemoveContainer" containerID="63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"
Jul 27 21:22:06.898908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.898892    2580 scope.go:117] "RemoveContainer" containerID="f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"
Jul 27 21:22:06.899161 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:06.899143    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13\": container with ID starting with f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13 not found: ID does not exist" containerID="f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"
Jul 27 21:22:06.899231 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"} err="failed to get container status \"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13\": rpc error: code = NotFound desc = could not find container \"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13\": container with ID starting with f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13 not found: ID does not exist"
Jul 27 21:22:06.899231 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899185    2580 scope.go:117] "RemoveContainer" containerID="63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"
Jul 27 21:22:06.899485 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:06.899458    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114\": container with ID starting with 63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114 not found: ID does not exist" containerID="63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"
Jul 27 21:22:06.899559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899493    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"} err="failed to get container status \"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114\": rpc error: code = NotFound desc = could not find container \"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114\": container with ID starting with 63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114 not found: ID does not exist"
Jul 27 21:22:06.899559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899516    2580 scope.go:117] "RemoveContainer" containerID="f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"
Jul 27 21:22:06.899747 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899723    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13"} err="failed to get container status \"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13\": rpc error: code = NotFound desc = could not find container \"f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13\": container with ID starting with f128da00130ec7da4cf2fc442939254fb508f4491e0423fbd6db7269d0670f13 not found: ID does not exist"
Jul 27 21:22:06.899810 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899748    2580 scope.go:117] "RemoveContainer" containerID="63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"
Jul 27 21:22:06.899918 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.899902    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114"} err="failed to get container status \"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114\": rpc error: code = NotFound desc = could not find container \"63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114\": container with ID starting with 63275122b24f188300c7b724fd9d648361c10f619c681b6eb3ec171a59e22114 not found: ID does not exist"
Jul 27 21:22:06.945141 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.945078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:22:06.948526 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:06.948502    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn"]
Jul 27 21:22:07.877840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:07.877811    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" path="/var/lib/kubelet/pods/78a0e727-f04a-4aad-b1c2-c9e018041cda/volumes"
Jul 27 21:22:08.476528 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:08.476483    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:10.903859 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:10.903828    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerID="3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0" exitCode=0
Jul 27 21:22:10.904290 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:10.903908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerDied","Data":"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"}
Jul 27 21:22:10.906439 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:10.906413    2580 generic.go:356] "Generic (PLEG): container finished" podID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerID="cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9" exitCode=0
Jul 27 21:22:10.906540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:10.906482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerDied","Data":"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"}
Jul 27 21:22:11.913301 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.913249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerStarted","Data":"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"}
Jul 27 21:22:11.913680 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.913327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerStarted","Data":"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"}
Jul 27 21:22:11.913680 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.913600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:11.913756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.913728    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:11.915119 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.915094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:11.933753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:11.933711    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podStartSLOduration=3.300108432 podStartE2EDuration="6.933698596s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633590164s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:11.932185925 +0000 UTC m=+1060.596811175" watchObservedRunningTime="2026-07-27 21:22:11.933698596 +0000 UTC m=+1060.598323833"
Jul 27 21:22:12.917552 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:12.917512    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:13.476091 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:13.476049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:13.480351 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:13.480328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:22:13.480728 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:13.480702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:22:17.921743 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:17.921717    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:22:17.922135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:17.922117    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:18.476383 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:18.476341    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:18.476579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:18.476471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:23.476229 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:23.476145    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:23.480796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:23.480764    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:22:23.481120 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:23.481097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:22:27.922777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:27.922738    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:28.475802 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:28.475762    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:33.476317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:33.476252    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:22:33.480672 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:33.480648    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:22:33.480817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:33.480802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:33.481054 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:33.481030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:22:33.481140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:33.481130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:35.829236 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.829207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:35.939332 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls\") pod \"7613dc26-bb6d-437b-85b1-a503350cbbdd\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") "
Jul 27 21:22:35.939463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert\") pod \"7613dc26-bb6d-437b-85b1-a503350cbbdd\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") "
Jul 27 21:22:35.939463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config\") pod \"7613dc26-bb6d-437b-85b1-a503350cbbdd\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") "
Jul 27 21:22:35.939463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location\") pod \"7613dc26-bb6d-437b-85b1-a503350cbbdd\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") "
Jul 27 21:22:35.939591 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmvpc\" (UniqueName: \"kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc\") pod \"7613dc26-bb6d-437b-85b1-a503350cbbdd\" (UID: \"7613dc26-bb6d-437b-85b1-a503350cbbdd\") "
Jul 27 21:22:35.939769 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7613dc26-bb6d-437b-85b1-a503350cbbdd" (UID: "7613dc26-bb6d-437b-85b1-a503350cbbdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:35.939769 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7613dc26-bb6d-437b-85b1-a503350cbbdd" (UID: "7613dc26-bb6d-437b-85b1-a503350cbbdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:35.940130 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.939773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7613dc26-bb6d-437b-85b1-a503350cbbdd" (UID: "7613dc26-bb6d-437b-85b1-a503350cbbdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:35.941549 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.941529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7613dc26-bb6d-437b-85b1-a503350cbbdd" (UID: "7613dc26-bb6d-437b-85b1-a503350cbbdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:35.941704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:35.941632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc" (OuterVolumeSpecName: "kube-api-access-kmvpc") pod "7613dc26-bb6d-437b-85b1-a503350cbbdd" (UID: "7613dc26-bb6d-437b-85b1-a503350cbbdd"). InnerVolumeSpecName "kube-api-access-kmvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:36.021407 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.021375    2580 generic.go:356] "Generic (PLEG): container finished" podID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerID="239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc" exitCode=0
Jul 27 21:22:36.021559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.021433    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"
Jul 27 21:22:36.021559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.021451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerDied","Data":"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"}
Jul 27 21:22:36.021559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.021527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf" event={"ID":"7613dc26-bb6d-437b-85b1-a503350cbbdd","Type":"ContainerDied","Data":"70604c2fccc1ba28f505935443f09f3b90479da6c596e978fd2cd519ed57d0d3"}
Jul 27 21:22:36.021680 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.021573    2580 scope.go:117] "RemoveContainer" containerID="239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"
Jul 27 21:22:36.030482 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.030466    2580 scope.go:117] "RemoveContainer" containerID="1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"
Jul 27 21:22:36.037665 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.037651    2580 scope.go:117] "RemoveContainer" containerID="cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"
Jul 27 21:22:36.040817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.040797    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7613dc26-bb6d-437b-85b1-a503350cbbdd-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:36.040817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.040819    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvpc\" (UniqueName: \"kubernetes.io/projected/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-api-access-kmvpc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:36.040937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.040828    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7613dc26-bb6d-437b-85b1-a503350cbbdd-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:36.040937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.040837    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:36.040937 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.040845    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7613dc26-bb6d-437b-85b1-a503350cbbdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:22:36.042793 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.042772    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:22:36.046740 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.046709    2580 scope.go:117] "RemoveContainer" containerID="54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"
Jul 27 21:22:36.046803 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.046786    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf"]
Jul 27 21:22:36.053837 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.053821    2580 scope.go:117] "RemoveContainer" containerID="239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"
Jul 27 21:22:36.054099 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:36.054081    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc\": container with ID starting with 239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc not found: ID does not exist" containerID="239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"
Jul 27 21:22:36.054160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054107    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc"} err="failed to get container status \"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc\": rpc error: code = NotFound desc = could not find container \"239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc\": container with ID starting with 239da88d66bc42e28d9992d271bd7a347296c666090ecc911ab277bc6ad960bc not found: ID does not exist"
Jul 27 21:22:36.054160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054124    2580 scope.go:117] "RemoveContainer" containerID="1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"
Jul 27 21:22:36.054425 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:36.054407    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344\": container with ID starting with 1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344 not found: ID does not exist" containerID="1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"
Jul 27 21:22:36.054476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054430    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344"} err="failed to get container status \"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344\": rpc error: code = NotFound desc = could not find container \"1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344\": container with ID starting with 1b520c6711dae4247ad27b4efd85b46351a417915f01aa0868b224dfd0bec344 not found: ID does not exist"
Jul 27 21:22:36.054476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054445    2580 scope.go:117] "RemoveContainer" containerID="cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"
Jul 27 21:22:36.054650 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:36.054634    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9\": container with ID starting with cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9 not found: ID does not exist" containerID="cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"
Jul 27 21:22:36.054693 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054655    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9"} err="failed to get container status \"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9\": rpc error: code = NotFound desc = could not find container \"cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9\": container with ID starting with cc28c049e3711577f61cf3b3408a9bbee4766b7479c1ed3d8d56bb4686f7a7b9 not found: ID does not exist"
Jul 27 21:22:36.054693 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054669    2580 scope.go:117] "RemoveContainer" containerID="54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"
Jul 27 21:22:36.054877 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:22:36.054860    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5\": container with ID starting with 54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5 not found: ID does not exist" containerID="54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"
Jul 27 21:22:36.054929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:36.054886    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5"} err="failed to get container status \"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5\": rpc error: code = NotFound desc = could not find container \"54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5\": container with ID starting with 54724ea08fae8cbb6915e5c0580b024f3207d004780b7e286b3a4dd0900019a5 not found: ID does not exist"
Jul 27 21:22:37.877836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:37.877803    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" path="/var/lib/kubelet/pods/7613dc26-bb6d-437b-85b1-a503350cbbdd/volumes"
Jul 27 21:22:37.922627 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:37.922596    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:47.923051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:47.923009    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:22:57.922385 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:22:57.922347    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:07.922462 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:07.922429    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:17.922309 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:17.922242    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:27.922518 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:27.922476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:37.922499 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:37.922457    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:47.872459 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:47.872405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:23:57.872501 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:23:57.872408    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:24:07.872617 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:07.872574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:24:17.872693 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:17.872646    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:24:27.872924 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:27.872885    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:24:37.878009 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:37.877975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:24:45.895607 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:45.895573    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:24:45.898111 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:45.895835    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" containerID="cri-o://bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6" gracePeriod=30
Jul 27 21:24:45.898111 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:45.895874    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy" containerID="cri-o://c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed" gracePeriod=30
Jul 27 21:24:46.617567 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:46.617536    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerID="c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed" exitCode=2
Jul 27 21:24:46.617749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:46.617608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerDied","Data":"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"}
Jul 27 21:24:47.873006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:47.872964    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:24:47.918463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:47.918431    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 27 21:24:51.073901 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.073865    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074203    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kserve-container"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074214    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kserve-container"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074231    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074236    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074243    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074276    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074285    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074295 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074297    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074304    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074318    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="storage-initializer"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074323    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="storage-initializer"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074379    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kserve-container"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074386    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074392    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="agent"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074399    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7613dc26-bb6d-437b-85b1-a503350cbbdd" containerName="kube-rbac-proxy"
Jul 27 21:24:51.074630 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.074406    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a0e727-f04a-4aad-b1c2-c9e018041cda" containerName="kserve-container"
Jul 27 21:24:51.077548 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.077530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.080016 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.079990    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:24:51.080431 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.080415    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 27 21:24:51.080723 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.080702    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 27 21:24:51.083440 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.083422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.085619 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.085600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 27 21:24:51.087121 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.087101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:24:51.095344 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.095323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:24:51.232126 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.232307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bld\" (UniqueName: \"kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.232307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7kk\" (UniqueName: \"kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.232307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.232307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.232307 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.232525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.232525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.232525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.232525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.232439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.333717 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.333717 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bld\" (UniqueName: \"kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.333717 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7kk\" (UniqueName: \"kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.333953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.333946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.334327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.334392 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.334492 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.334590 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.334812 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.334868 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.334846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.336502 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.336473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.336598 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.336512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.341479 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.341457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7kk\" (UniqueName: \"kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk\") pod \"isvc-canary-create-predictor-cb85cf55c-vzbc6\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.341677 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.341660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bld\" (UniqueName: \"kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.388645 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.388622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:51.397764 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.397736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:51.531204 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.531181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:24:51.533211 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:24:51.533182    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83edc869_5d06_446f_90ee_a33f424af082.slice/crio-07f8584ad342519fb7bbadd5e93ac93efc1c5e52227a69b5b27b4d474cb82bc1 WatchSource:0}: Error finding container 07f8584ad342519fb7bbadd5e93ac93efc1c5e52227a69b5b27b4d474cb82bc1: Status 404 returned error can't find the container with id 07f8584ad342519fb7bbadd5e93ac93efc1c5e52227a69b5b27b4d474cb82bc1
Jul 27 21:24:51.553461 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.553439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:24:51.555311 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:24:51.555287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51899645_9f54_4b8f_a0dc_62d5263e7952.slice/crio-ef1d74065e6aa7839f4a1d37901af83ebb408662aeca007a1058539e98c2b244 WatchSource:0}: Error finding container ef1d74065e6aa7839f4a1d37901af83ebb408662aeca007a1058539e98c2b244: Status 404 returned error can't find the container with id ef1d74065e6aa7839f4a1d37901af83ebb408662aeca007a1058539e98c2b244
Jul 27 21:24:51.642620 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.642573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerStarted","Data":"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"}
Jul 27 21:24:51.642781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.642647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerStarted","Data":"07f8584ad342519fb7bbadd5e93ac93efc1c5e52227a69b5b27b4d474cb82bc1"}
Jul 27 21:24:51.644386 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.644348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerStarted","Data":"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"}
Jul 27 21:24:51.644501 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:51.644405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerStarted","Data":"ef1d74065e6aa7839f4a1d37901af83ebb408662aeca007a1058539e98c2b244"}
Jul 27 21:24:52.918507 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:52.918468    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 27 21:24:55.346451 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.346427    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:24:55.469758 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.469729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert\") pod \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") "
Jul 27 21:24:55.469886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.469763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttfj7\" (UniqueName: \"kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7\") pod \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") "
Jul 27 21:24:55.469886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.469795    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config\") pod \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") "
Jul 27 21:24:55.469886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.469816    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location\") pod \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") "
Jul 27 21:24:55.469886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.469867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") pod \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\" (UID: \"a8b61230-766e-42bb-af1d-b1a7b0d8bab4\") "
Jul 27 21:24:55.470165 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.470135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8b61230-766e-42bb-af1d-b1a7b0d8bab4" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:55.470286 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.470240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8b61230-766e-42bb-af1d-b1a7b0d8bab4" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:55.470460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.470435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8b61230-766e-42bb-af1d-b1a7b0d8bab4" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:55.472031 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.472010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7" (OuterVolumeSpecName: "kube-api-access-ttfj7") pod "a8b61230-766e-42bb-af1d-b1a7b0d8bab4" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4"). InnerVolumeSpecName "kube-api-access-ttfj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:55.472104 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.472090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8b61230-766e-42bb-af1d-b1a7b0d8bab4" (UID: "a8b61230-766e-42bb-af1d-b1a7b0d8bab4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:24:55.570584 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.570550    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:24:55.570584 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.570580    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:24:55.570584 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.570592    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:24:55.570747 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.570601    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:24:55.570747 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.570610    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttfj7\" (UniqueName: \"kubernetes.io/projected/a8b61230-766e-42bb-af1d-b1a7b0d8bab4-kube-api-access-ttfj7\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:24:55.665531 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.665497    2580 generic.go:356] "Generic (PLEG): container finished" podID="83edc869-5d06-446f-90ee-a33f424af082" containerID="845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd" exitCode=0
Jul 27 21:24:55.665673 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.665575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerDied","Data":"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"}
Jul 27 21:24:55.667484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.667466    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerID="bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6" exitCode=0
Jul 27 21:24:55.667577 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.667532    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"
Jul 27 21:24:55.667577 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.667541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerDied","Data":"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"}
Jul 27 21:24:55.667724 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.667593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv" event={"ID":"a8b61230-766e-42bb-af1d-b1a7b0d8bab4","Type":"ContainerDied","Data":"ba169d0e134b41ed23459dadccbae608e9534580b10b58261c4a9a6ba23a8701"}
Jul 27 21:24:55.667724 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.667624    2580 scope.go:117] "RemoveContainer" containerID="c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"
Jul 27 21:24:55.669247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.669214    2580 generic.go:356] "Generic (PLEG): container finished" podID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerID="e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71" exitCode=0
Jul 27 21:24:55.669355 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.669292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerDied","Data":"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"}
Jul 27 21:24:55.678855 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.678842    2580 scope.go:117] "RemoveContainer" containerID="bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"
Jul 27 21:24:55.688915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.688895    2580 scope.go:117] "RemoveContainer" containerID="3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"
Jul 27 21:24:55.707362 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.707341    2580 scope.go:117] "RemoveContainer" containerID="c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"
Jul 27 21:24:55.707709 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:24:55.707685    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed\": container with ID starting with c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed not found: ID does not exist" containerID="c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"
Jul 27 21:24:55.707765 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.707722    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed"} err="failed to get container status \"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed\": rpc error: code = NotFound desc = could not find container \"c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed\": container with ID starting with c622f1197df127459209c231616cdc9384e1cd43d9a15bb9fe5f6e032b6f0eed not found: ID does not exist"
Jul 27 21:24:55.707765 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.707746    2580 scope.go:117] "RemoveContainer" containerID="bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"
Jul 27 21:24:55.708027 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:24:55.708008    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6\": container with ID starting with bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6 not found: ID does not exist" containerID="bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"
Jul 27 21:24:55.708093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.708033    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6"} err="failed to get container status \"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6\": rpc error: code = NotFound desc = could not find container \"bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6\": container with ID starting with bfd00060725175871faf565d2a592f4ce6642577d8c09756ee1f1e35a81b24b6 not found: ID does not exist"
Jul 27 21:24:55.708093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.708049    2580 scope.go:117] "RemoveContainer" containerID="3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"
Jul 27 21:24:55.708335 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:24:55.708308    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0\": container with ID starting with 3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0 not found: ID does not exist" containerID="3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"
Jul 27 21:24:55.708441 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.708342    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0"} err="failed to get container status \"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0\": rpc error: code = NotFound desc = could not find container \"3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0\": container with ID starting with 3f354039a4da531a1a27bbf662ae7bfaadddc47f97866e46df3a45fa5b266fc0 not found: ID does not exist"
Jul 27 21:24:55.713954 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.713929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:24:55.715621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.715605    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv"]
Jul 27 21:24:55.878488 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:55.878461    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" path="/var/lib/kubelet/pods/a8b61230-766e-42bb-af1d-b1a7b0d8bab4/volumes"
Jul 27 21:24:56.678201 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.678165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerStarted","Data":"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"}
Jul 27 21:24:56.678761 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.678235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerStarted","Data":"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"}
Jul 27 21:24:56.678761 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.678555    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:56.678761 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.678708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:24:56.680245 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.680218    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:24:56.681195 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.681176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerStarted","Data":"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"}
Jul 27 21:24:56.681325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.681224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerStarted","Data":"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"}
Jul 27 21:24:56.681421 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.681408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:56.698003 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.697965    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podStartSLOduration=2.122754864 podStartE2EDuration="5.697951948s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575197084s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:56.6966741 +0000 UTC m=+1225.361299339" watchObservedRunningTime="2026-07-27 21:24:56.697951948 +0000 UTC m=+1225.362577185"
Jul 27 21:24:56.720029 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:56.719989    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podStartSLOduration=1.86047003 podStartE2EDuration="5.719971823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859501793s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:56.718127697 +0000 UTC m=+1225.382752962" watchObservedRunningTime="2026-07-27 21:24:56.719971823 +0000 UTC m=+1225.384597061"
Jul 27 21:24:57.686554 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:57.686511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:24:57.686926 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:57.686573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:24:57.687670 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:57.687638    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:24:58.691229 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:24:58.691194    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:02.690692 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:02.690659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:25:02.691198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:02.691174    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:03.695673 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:03.695646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:25:03.696315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:03.696287    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:12.691805 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:12.691765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:13.697232 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:13.697197    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:22.691642 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:22.691561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:23.696545 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:23.696507    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:32.691650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:32.691613    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:33.696460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:33.696424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:42.691750 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:42.691710    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:43.696274 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:43.696232    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:25:52.691427 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:52.691383    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:25:53.697084 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:25:53.697047    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:26:02.691637 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:02.691594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:26:03.696304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:03.696249    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:26:12.691423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:12.691392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:26:13.696975 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:13.696941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:26:21.155535 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.155493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:26:21.155946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.155810    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" containerID="cri-o://c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34" gracePeriod=30
Jul 27 21:26:21.155946 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.155842    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kube-rbac-proxy" containerID="cri-o://34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11" gracePeriod=30
Jul 27 21:26:21.160604 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.160575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:26:21.160862 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.160837    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" containerID="cri-o://4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18" gracePeriod=30
Jul 27 21:26:21.160940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:21.160857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kube-rbac-proxy" containerID="cri-o://842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8" gracePeriod=30
Jul 27 21:26:22.077082 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.077044    2580 generic.go:356] "Generic (PLEG): container finished" podID="83edc869-5d06-446f-90ee-a33f424af082" containerID="842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8" exitCode=2
Jul 27 21:26:22.077282 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.077126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerDied","Data":"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"}
Jul 27 21:26:22.079159 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.079139    2580 generic.go:356] "Generic (PLEG): container finished" podID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerID="34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11" exitCode=2
Jul 27 21:26:22.079309 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.079208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerDied","Data":"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"}
Jul 27 21:26:22.686647 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.686615    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 27 21:26:22.691957 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:22.691936    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:26:23.692356 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:23.692318    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 27 21:26:23.696646 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:23.696627    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:26:25.830854 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.830823    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:26:25.834860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.834835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:26:25.907862 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.907841    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q7kk\" (UniqueName: \"kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk\") pod \"83edc869-5d06-446f-90ee-a33f424af082\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") "
Jul 27 21:26:25.908017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.907870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2bld\" (UniqueName: \"kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld\") pod \"51899645-9f54-4b8f-a0dc-62d5263e7952\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") "
Jul 27 21:26:25.908017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.907913    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location\") pod \"83edc869-5d06-446f-90ee-a33f424af082\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") "
Jul 27 21:26:25.908140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert\") pod \"51899645-9f54-4b8f-a0dc-62d5263e7952\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") "
Jul 27 21:26:25.908198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location\") pod \"51899645-9f54-4b8f-a0dc-62d5263e7952\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") "
Jul 27 21:26:25.908198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config\") pod \"51899645-9f54-4b8f-a0dc-62d5263e7952\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") "
Jul 27 21:26:25.908198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config\") pod \"83edc869-5d06-446f-90ee-a33f424af082\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") "
Jul 27 21:26:25.908394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert\") pod \"83edc869-5d06-446f-90ee-a33f424af082\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") "
Jul 27 21:26:25.908394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls\") pod \"83edc869-5d06-446f-90ee-a33f424af082\" (UID: \"83edc869-5d06-446f-90ee-a33f424af082\") "
Jul 27 21:26:25.908394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83edc869-5d06-446f-90ee-a33f424af082" (UID: "83edc869-5d06-446f-90ee-a33f424af082"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:25.908394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls\") pod \"51899645-9f54-4b8f-a0dc-62d5263e7952\" (UID: \"51899645-9f54-4b8f-a0dc-62d5263e7952\") "
Jul 27 21:26:25.908394 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51899645-9f54-4b8f-a0dc-62d5263e7952" (UID: "51899645-9f54-4b8f-a0dc-62d5263e7952"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:25.908807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51899645-9f54-4b8f-a0dc-62d5263e7952" (UID: "51899645-9f54-4b8f-a0dc-62d5263e7952"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:25.908807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908623    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83edc869-5d06-446f-90ee-a33f424af082-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:25.908807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908651    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:25.908807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83edc869-5d06-446f-90ee-a33f424af082" (UID: "83edc869-5d06-446f-90ee-a33f424af082"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:25.908807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908767    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83edc869-5d06-446f-90ee-a33f424af082" (UID: "83edc869-5d06-446f-90ee-a33f424af082"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:25.909086 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.908998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51899645-9f54-4b8f-a0dc-62d5263e7952" (UID: "51899645-9f54-4b8f-a0dc-62d5263e7952"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:25.910242 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.910218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld" (OuterVolumeSpecName: "kube-api-access-c2bld") pod "51899645-9f54-4b8f-a0dc-62d5263e7952" (UID: "51899645-9f54-4b8f-a0dc-62d5263e7952"). InnerVolumeSpecName "kube-api-access-c2bld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:25.910359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.910270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk" (OuterVolumeSpecName: "kube-api-access-6q7kk") pod "83edc869-5d06-446f-90ee-a33f424af082" (UID: "83edc869-5d06-446f-90ee-a33f424af082"). InnerVolumeSpecName "kube-api-access-6q7kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:25.910475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.910459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83edc869-5d06-446f-90ee-a33f424af082" (UID: "83edc869-5d06-446f-90ee-a33f424af082"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:25.910617 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:25.910605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51899645-9f54-4b8f-a0dc-62d5263e7952" (UID: "51899645-9f54-4b8f-a0dc-62d5263e7952"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009094    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51899645-9f54-4b8f-a0dc-62d5263e7952-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009115    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009125    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009135    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83edc869-5d06-446f-90ee-a33f424af082-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009145    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83edc869-5d06-446f-90ee-a33f424af082-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009152    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51899645-9f54-4b8f-a0dc-62d5263e7952-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009161    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q7kk\" (UniqueName: \"kubernetes.io/projected/83edc869-5d06-446f-90ee-a33f424af082-kube-api-access-6q7kk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.009454 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.009169    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bld\" (UniqueName: \"kubernetes.io/projected/51899645-9f54-4b8f-a0dc-62d5263e7952-kube-api-access-c2bld\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:26:26.098579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.098546    2580 generic.go:356] "Generic (PLEG): container finished" podID="83edc869-5d06-446f-90ee-a33f424af082" containerID="4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18" exitCode=0
Jul 27 21:26:26.098720 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.098601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"
Jul 27 21:26:26.098720 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.098630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerDied","Data":"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"}
Jul 27 21:26:26.098720 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.098692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6" event={"ID":"83edc869-5d06-446f-90ee-a33f424af082","Type":"ContainerDied","Data":"07f8584ad342519fb7bbadd5e93ac93efc1c5e52227a69b5b27b4d474cb82bc1"}
Jul 27 21:26:26.098858 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.098728    2580 scope.go:117] "RemoveContainer" containerID="842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"
Jul 27 21:26:26.100636 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.100611    2580 generic.go:356] "Generic (PLEG): container finished" podID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerID="c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34" exitCode=0
Jul 27 21:26:26.100737 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.100655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"
Jul 27 21:26:26.100781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.100653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerDied","Data":"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"}
Jul 27 21:26:26.100817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.100792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6" event={"ID":"51899645-9f54-4b8f-a0dc-62d5263e7952","Type":"ContainerDied","Data":"ef1d74065e6aa7839f4a1d37901af83ebb408662aeca007a1058539e98c2b244"}
Jul 27 21:26:26.108700 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.108683    2580 scope.go:117] "RemoveContainer" containerID="4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"
Jul 27 21:26:26.116631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.116616    2580 scope.go:117] "RemoveContainer" containerID="845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"
Jul 27 21:26:26.122926 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.122905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:26:26.124876 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.124859    2580 scope.go:117] "RemoveContainer" containerID="842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"
Jul 27 21:26:26.125248 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.125213    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8\": container with ID starting with 842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8 not found: ID does not exist" containerID="842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"
Jul 27 21:26:26.125417 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.125249    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8"} err="failed to get container status \"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8\": rpc error: code = NotFound desc = could not find container \"842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8\": container with ID starting with 842e2bc8f22b3635de1aa6c897cb9aa575cc7913f743b21b351c053ade820de8 not found: ID does not exist"
Jul 27 21:26:26.125521 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.125420    2580 scope.go:117] "RemoveContainer" containerID="4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"
Jul 27 21:26:26.125758 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.125735    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18\": container with ID starting with 4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18 not found: ID does not exist" containerID="4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"
Jul 27 21:26:26.125826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.125767    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18"} err="failed to get container status \"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18\": rpc error: code = NotFound desc = could not find container \"4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18\": container with ID starting with 4056240af1fe8017de87d7a892a7315711375266c3bf74febf06bd4a768c7e18 not found: ID does not exist"
Jul 27 21:26:26.125826 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.125788    2580 scope.go:117] "RemoveContainer" containerID="845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"
Jul 27 21:26:26.126054 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.126037    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd\": container with ID starting with 845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd not found: ID does not exist" containerID="845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"
Jul 27 21:26:26.126107 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.126067    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd"} err="failed to get container status \"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd\": rpc error: code = NotFound desc = could not find container \"845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd\": container with ID starting with 845872bad8f2c35d94ca556234c5d1060aac99870432072a2c47a591bf15e7dd not found: ID does not exist"
Jul 27 21:26:26.126107 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.126095    2580 scope.go:117] "RemoveContainer" containerID="34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"
Jul 27 21:26:26.127582 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.127563    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6"]
Jul 27 21:26:26.133538 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.133523    2580 scope.go:117] "RemoveContainer" containerID="c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"
Jul 27 21:26:26.139967 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.139946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:26:26.141927 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.141911    2580 scope.go:117] "RemoveContainer" containerID="e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"
Jul 27 21:26:26.143464 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.143447    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6"]
Jul 27 21:26:26.149460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.149445    2580 scope.go:117] "RemoveContainer" containerID="34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"
Jul 27 21:26:26.149689 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.149671    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11\": container with ID starting with 34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11 not found: ID does not exist" containerID="34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"
Jul 27 21:26:26.149735 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.149706    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11"} err="failed to get container status \"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11\": rpc error: code = NotFound desc = could not find container \"34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11\": container with ID starting with 34b4ed8873a8540496ee683cd691d61c0ee95dec88081b0fd5181aa59b9b5b11 not found: ID does not exist"
Jul 27 21:26:26.149735 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.149725    2580 scope.go:117] "RemoveContainer" containerID="c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"
Jul 27 21:26:26.149907 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.149893    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34\": container with ID starting with c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34 not found: ID does not exist" containerID="c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"
Jul 27 21:26:26.149955 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.149911    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34"} err="failed to get container status \"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34\": rpc error: code = NotFound desc = could not find container \"c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34\": container with ID starting with c56dbba43caf9fe72aa7fcfcf4666dad98f8b428e1be44b82c28204ecc7e8e34 not found: ID does not exist"
Jul 27 21:26:26.149955 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.149923    2580 scope.go:117] "RemoveContainer" containerID="e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"
Jul 27 21:26:26.150099 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.150077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71\": container with ID starting with e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71 not found: ID does not exist" containerID="e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"
Jul 27 21:26:26.150190 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.150108    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71"} err="failed to get container status \"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71\": rpc error: code = NotFound desc = could not find container \"e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71\": container with ID starting with e71b58f6902b8c4d47d86886482e503ee4ce97a0b4fb89c00c4fe8a9144c2e71 not found: ID does not exist"
Jul 27 21:26:26.309078 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309043    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:26:26.309403 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309390    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309405    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309416    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309421    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309431    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309437    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309444    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="storage-initializer"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="storage-initializer"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309454    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="storage-initializer"
Jul 27 21:26:26.309463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309460    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="storage-initializer"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309473    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309481    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309486    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309498    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="storage-initializer"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309503    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="storage-initializer"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309516    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309564    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309573    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309580    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b61230-766e-42bb-af1d-b1a7b0d8bab4" containerName="kube-rbac-proxy"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309591    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kserve-container"
Jul 27 21:26:26.309726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.309597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83edc869-5d06-446f-90ee-a33f424af082" containerName="kube-rbac-proxy"
Jul 27 21:26:26.314796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.314779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:26:26.314913 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.314897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.317418 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 27 21:26:26.317674 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:26:26.317787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:26:26.317787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:26:26.317787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:26:26.317787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:26:26.317965 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.317953    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 27 21:26:26.318056 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.318040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.320338 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.320322    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 27 21:26:26.324502 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.324483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:26:26.329550 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.329525    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:26:26.412993 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.412969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.413122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5447x\" (UniqueName: \"kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.413122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.413198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.413198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.413198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.413334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.413334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.413334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbwq\" (UniqueName: \"kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.413334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.413327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514044 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5447x\" (UniqueName: \"kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514366 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.514627 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbwq\" (UniqueName: \"kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514627 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.514429    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 27 21:26:26.514627 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:26:26.514494    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls podName:28293c57-6cb3-4bea-80b2-9b71322a8cc4 nodeName:}" failed. No retries permitted until 2026-07-27 21:26:27.014475756 +0000 UTC m=+1315.679100990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls") pod "isvc-canary-promote-predictor-d9759ffd5-ppssc" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 27 21:26:26.514850 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514913 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.514913 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.515010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.515010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.515010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.514942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.516941 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.516918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.523298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.523244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbwq\" (UniqueName: \"kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:26.523397 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.523307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5447x\" (UniqueName: \"kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x\") pod \"isvc-canary-promote-v2-predictor-86766c4958-sz22f\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.634782 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.634726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:26.768873 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:26.768847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:26:26.770924 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:26:26.770899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc1a46_1cf6_4a07_b505_02fb14224a61.slice/crio-8d7b73572e95277550738e93de27e773393c8f3b40cea74e96d1f088ce449e4d WatchSource:0}: Error finding container 8d7b73572e95277550738e93de27e773393c8f3b40cea74e96d1f088ce449e4d: Status 404 returned error can't find the container with id 8d7b73572e95277550738e93de27e773393c8f3b40cea74e96d1f088ce449e4d
Jul 27 21:26:27.019145 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.019061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:27.021515 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.021499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppssc\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:27.106635 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.106595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerStarted","Data":"6edd1fac87caee978cd6fd38b1150d74b7dfcf67b9429516f42285a3e73c7251"}
Jul 27 21:26:27.106815 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.106668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerStarted","Data":"8d7b73572e95277550738e93de27e773393c8f3b40cea74e96d1f088ce449e4d"}
Jul 27 21:26:27.227874 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.227840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:27.369078 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.369054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:26:27.370729 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:26:27.370707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28293c57_6cb3_4bea_80b2_9b71322a8cc4.slice/crio-78eeda0dda56b9d5ba7c551e3222b57a2a173aedfb7b5b61b5a6c1e61fa3ea6d WatchSource:0}: Error finding container 78eeda0dda56b9d5ba7c551e3222b57a2a173aedfb7b5b61b5a6c1e61fa3ea6d: Status 404 returned error can't find the container with id 78eeda0dda56b9d5ba7c551e3222b57a2a173aedfb7b5b61b5a6c1e61fa3ea6d
Jul 27 21:26:27.878430 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.878399    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51899645-9f54-4b8f-a0dc-62d5263e7952" path="/var/lib/kubelet/pods/51899645-9f54-4b8f-a0dc-62d5263e7952/volumes"
Jul 27 21:26:27.878881 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:27.878868    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83edc869-5d06-446f-90ee-a33f424af082" path="/var/lib/kubelet/pods/83edc869-5d06-446f-90ee-a33f424af082/volumes"
Jul 27 21:26:28.114341 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:28.114305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerStarted","Data":"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"}
Jul 27 21:26:28.114760 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:28.114362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerStarted","Data":"78eeda0dda56b9d5ba7c551e3222b57a2a173aedfb7b5b61b5a6c1e61fa3ea6d"}
Jul 27 21:26:31.130379 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:31.130346    2580 generic.go:356] "Generic (PLEG): container finished" podID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerID="6edd1fac87caee978cd6fd38b1150d74b7dfcf67b9429516f42285a3e73c7251" exitCode=0
Jul 27 21:26:31.130781 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:31.130418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerDied","Data":"6edd1fac87caee978cd6fd38b1150d74b7dfcf67b9429516f42285a3e73c7251"}
Jul 27 21:26:31.132022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:31.132003    2580 generic.go:356] "Generic (PLEG): container finished" podID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerID="ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f" exitCode=0
Jul 27 21:26:31.132118 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:31.132058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerDied","Data":"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"}
Jul 27 21:26:32.138066 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.137979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerStarted","Data":"b2cf931c3bb139b47d9efbc4de138da7b6721860d984b5994699fa2c8e611d8c"}
Jul 27 21:26:32.138066 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.138047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerStarted","Data":"007fa55e3db155c1ec8f20af97fb35f9d78e4aa423ef4f9ff572494868ca5dc8"}
Jul 27 21:26:32.138587 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.138316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:32.138587 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.138380    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:32.140105 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.140075    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:26:32.140249 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.140231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerStarted","Data":"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"}
Jul 27 21:26:32.140342 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.140296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerStarted","Data":"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"}
Jul 27 21:26:32.140529 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.140514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:32.140597 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.140553    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:32.141639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.141611    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:26:32.157947 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.157904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podStartSLOduration=2.495688688 podStartE2EDuration="6.157892409s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662203721s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:32.156174339 +0000 UTC m=+1320.820799620" watchObservedRunningTime="2026-07-27 21:26:32.157892409 +0000 UTC m=+1320.822517717"
Jul 27 21:26:32.181440 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:32.181390    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podStartSLOduration=2.523977393 podStartE2EDuration="6.181375731s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657398338s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:32.179736049 +0000 UTC m=+1320.844361299" watchObservedRunningTime="2026-07-27 21:26:32.181375731 +0000 UTC m=+1320.846000970"
Jul 27 21:26:33.144768 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:33.144723    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:26:33.145199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:33.144846    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:26:38.149315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:38.149286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:26:38.149848 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:38.149671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:26:38.149848 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:38.149691    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:26:38.150041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:38.150022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:26:48.150598 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:48.150559    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:26:48.150989 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:48.150560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:26:58.150150 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:58.150074    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:26:58.150529 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:26:58.150074    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:27:08.149705 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:08.149661    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:27:08.150220 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:08.150091    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:27:18.150100 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:18.150059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:27:18.150598 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:18.150064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:27:28.150292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:28.150233    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:27:28.150292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:28.150233    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:27:38.150475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:38.150432    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:27:38.150905 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:38.150432    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:27:48.151352 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:48.151325    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:27:48.151758 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:48.151412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:27:56.375208 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:56.375172    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:27:56.375794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:56.375487    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" containerID="cri-o://24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1" gracePeriod=30
Jul 27 21:27:56.375794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:56.375556    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kube-rbac-proxy" containerID="cri-o://0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6" gracePeriod=30
Jul 27 21:27:56.534770 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:56.534740    2580 generic.go:356] "Generic (PLEG): container finished" podID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerID="0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6" exitCode=2
Jul 27 21:27:56.534888 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:56.534809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerDied","Data":"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"}
Jul 27 21:27:58.145149 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:58.145102    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 27 21:27:58.150283 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:27:58.150239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:28:00.523107 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.523085    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:28:00.555493 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.555457    2580 generic.go:356] "Generic (PLEG): container finished" podID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerID="24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1" exitCode=0
Jul 27 21:28:00.555628 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.555517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"
Jul 27 21:28:00.555628 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.555537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerDied","Data":"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"}
Jul 27 21:28:00.555628 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.555617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc" event={"ID":"28293c57-6cb3-4bea-80b2-9b71322a8cc4","Type":"ContainerDied","Data":"78eeda0dda56b9d5ba7c551e3222b57a2a173aedfb7b5b61b5a6c1e61fa3ea6d"}
Jul 27 21:28:00.555799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.555656    2580 scope.go:117] "RemoveContainer" containerID="0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"
Jul 27 21:28:00.564177 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.564151    2580 scope.go:117] "RemoveContainer" containerID="24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"
Jul 27 21:28:00.573249 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.573221    2580 scope.go:117] "RemoveContainer" containerID="ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"
Jul 27 21:28:00.584928 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.584909    2580 scope.go:117] "RemoveContainer" containerID="0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"
Jul 27 21:28:00.585217 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:28:00.585197    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6\": container with ID starting with 0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6 not found: ID does not exist" containerID="0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"
Jul 27 21:28:00.585317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.585232    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6"} err="failed to get container status \"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6\": rpc error: code = NotFound desc = could not find container \"0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6\": container with ID starting with 0bff9906e68e7ecf6c40f1f81ee7c96548f69fda8efdbec1da788ce5c5004ff6 not found: ID does not exist"
Jul 27 21:28:00.585401 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.585321    2580 scope.go:117] "RemoveContainer" containerID="24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"
Jul 27 21:28:00.585607 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:28:00.585589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1\": container with ID starting with 24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1 not found: ID does not exist" containerID="24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"
Jul 27 21:28:00.585661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.585614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1"} err="failed to get container status \"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1\": rpc error: code = NotFound desc = could not find container \"24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1\": container with ID starting with 24a5ec5972fdcd1d92f9f23502b6664af803cc62060cb32cc36676245cf07bc1 not found: ID does not exist"
Jul 27 21:28:00.585661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.585631    2580 scope.go:117] "RemoveContainer" containerID="ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"
Jul 27 21:28:00.585854 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:28:00.585834    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f\": container with ID starting with ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f not found: ID does not exist" containerID="ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"
Jul 27 21:28:00.585916 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.585865    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f"} err="failed to get container status \"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f\": rpc error: code = NotFound desc = could not find container \"ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f\": container with ID starting with ae0600d7ef38d590eaa516275b09f67acd6d793e1d7eb4f9bba0abc9ee0c363f not found: ID does not exist"
Jul 27 21:28:00.703006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.702908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") pod \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") "
Jul 27 21:28:00.703006 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.702973    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location\") pod \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") "
Jul 27 21:28:00.703237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703036    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config\") pod \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") "
Jul 27 21:28:00.703237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert\") pod \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") "
Jul 27 21:28:00.703237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhbwq\" (UniqueName: \"kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq\") pod \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\" (UID: \"28293c57-6cb3-4bea-80b2-9b71322a8cc4\") "
Jul 27 21:28:00.703432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28293c57-6cb3-4bea-80b2-9b71322a8cc4" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:28:00.703475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28293c57-6cb3-4bea-80b2-9b71322a8cc4" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:00.703475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.703457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28293c57-6cb3-4bea-80b2-9b71322a8cc4" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:00.705190 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.705170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28293c57-6cb3-4bea-80b2-9b71322a8cc4" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:28:00.705298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.705280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq" (OuterVolumeSpecName: "kube-api-access-xhbwq") pod "28293c57-6cb3-4bea-80b2-9b71322a8cc4" (UID: "28293c57-6cb3-4bea-80b2-9b71322a8cc4"). InnerVolumeSpecName "kube-api-access-xhbwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:00.804497 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.804461    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28293c57-6cb3-4bea-80b2-9b71322a8cc4-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:28:00.804497 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.804493    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:28:00.804683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.804506    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:28:00.804683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.804519    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28293c57-6cb3-4bea-80b2-9b71322a8cc4-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:28:00.804683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.804532    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbwq\" (UniqueName: \"kubernetes.io/projected/28293c57-6cb3-4bea-80b2-9b71322a8cc4-kube-api-access-xhbwq\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:28:00.877901 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.877872    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:28:00.881745 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:00.881711    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc"]
Jul 27 21:28:01.878268 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:28:01.878233    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" path="/var/lib/kubelet/pods/28293c57-6cb3-4bea-80b2-9b71322a8cc4/volumes"
Jul 27 21:29:36.609766 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:36.609730    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:29:36.610376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:36.610021    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" containerID="cri-o://007fa55e3db155c1ec8f20af97fb35f9d78e4aa423ef4f9ff572494868ca5dc8" gracePeriod=30
Jul 27 21:29:36.610376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:36.610096    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kube-rbac-proxy" containerID="cri-o://b2cf931c3bb139b47d9efbc4de138da7b6721860d984b5994699fa2c8e611d8c" gracePeriod=30
Jul 27 21:29:36.991775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:36.991685    2580 generic.go:356] "Generic (PLEG): container finished" podID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerID="b2cf931c3bb139b47d9efbc4de138da7b6721860d984b5994699fa2c8e611d8c" exitCode=2
Jul 27 21:29:36.991775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:36.991744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerDied","Data":"b2cf931c3bb139b47d9efbc4de138da7b6721860d984b5994699fa2c8e611d8c"}
Jul 27 21:29:38.145398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:38.145356    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 27 21:29:38.150225 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:38.150195    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 21:29:41.012904 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.012874    2580 generic.go:356] "Generic (PLEG): container finished" podID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerID="007fa55e3db155c1ec8f20af97fb35f9d78e4aa423ef4f9ff572494868ca5dc8" exitCode=0
Jul 27 21:29:41.013245 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.012941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerDied","Data":"007fa55e3db155c1ec8f20af97fb35f9d78e4aa423ef4f9ff572494868ca5dc8"}
Jul 27 21:29:41.060831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.060806    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:29:41.132485 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5447x\" (UniqueName: \"kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x\") pod \"cccc1a46-1cf6-4a07-b505-02fb14224a61\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") "
Jul 27 21:29:41.132485 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config\") pod \"cccc1a46-1cf6-4a07-b505-02fb14224a61\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") "
Jul 27 21:29:41.132707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls\") pod \"cccc1a46-1cf6-4a07-b505-02fb14224a61\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") "
Jul 27 21:29:41.132707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location\") pod \"cccc1a46-1cf6-4a07-b505-02fb14224a61\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") "
Jul 27 21:29:41.132707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert\") pod \"cccc1a46-1cf6-4a07-b505-02fb14224a61\" (UID: \"cccc1a46-1cf6-4a07-b505-02fb14224a61\") "
Jul 27 21:29:41.132904 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cccc1a46-1cf6-4a07-b505-02fb14224a61" (UID: "cccc1a46-1cf6-4a07-b505-02fb14224a61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:41.132970 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cccc1a46-1cf6-4a07-b505-02fb14224a61" (UID: "cccc1a46-1cf6-4a07-b505-02fb14224a61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:41.132970 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.132947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cccc1a46-1cf6-4a07-b505-02fb14224a61" (UID: "cccc1a46-1cf6-4a07-b505-02fb14224a61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:41.134760 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.134739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x" (OuterVolumeSpecName: "kube-api-access-5447x") pod "cccc1a46-1cf6-4a07-b505-02fb14224a61" (UID: "cccc1a46-1cf6-4a07-b505-02fb14224a61"). InnerVolumeSpecName "kube-api-access-5447x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:41.134825 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.134752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cccc1a46-1cf6-4a07-b505-02fb14224a61" (UID: "cccc1a46-1cf6-4a07-b505-02fb14224a61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:41.233894 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.233867    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5447x\" (UniqueName: \"kubernetes.io/projected/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-api-access-5447x\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:29:41.233894 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.233892    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:29:41.234041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.233906    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cccc1a46-1cf6-4a07-b505-02fb14224a61-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:29:41.234041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.233923    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cccc1a46-1cf6-4a07-b505-02fb14224a61-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:29:41.234041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.233935    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cccc1a46-1cf6-4a07-b505-02fb14224a61-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:29:41.771462 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771428    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:29:41.771790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771777    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="storage-initializer"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771792    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="storage-initializer"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771805    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="storage-initializer"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771811    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="storage-initializer"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771820    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kube-rbac-proxy"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771826    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kube-rbac-proxy"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container"
Jul 27 21:29:41.771842 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771855    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771860    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771865    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kube-rbac-proxy"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771870    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kube-rbac-proxy"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771923    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kube-rbac-proxy"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771929    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kserve-container"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771937    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" containerName="kserve-container"
Jul 27 21:29:41.772087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.771944    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28293c57-6cb3-4bea-80b2-9b71322a8cc4" containerName="kube-rbac-proxy"
Jul 27 21:29:41.775461 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.775443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.778009 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.777989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 27 21:29:41.778087 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.777989    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 27 21:29:41.782001 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.781978    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:29:41.785554 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.785536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.787936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.787919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 27 21:29:41.789043 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.789019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:29:41.795239 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.795214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:29:41.839103 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.839247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.839247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.839247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.839247 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.839449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfj4\" (UniqueName: \"kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.839449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqnx\" (UniqueName: \"kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.839449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.839449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.839449 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.839396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940473 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.940473 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.940734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940734 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:29:41.940588    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 27 21:29:41.940734 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.940734 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:29:41.940671    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls podName:991b44d7-0c58-4bbc-9619-397862744783 nodeName:}" failed. No retries permitted until 2026-07-27 21:29:42.440648213 +0000 UTC m=+1511.105273430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" (UID: "991b44d7-0c58-4bbc-9619-397862744783") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfj4\" (UniqueName: \"kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:29:41.940924    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 27 21:29:41.940956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqnx\" (UniqueName: \"kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.941311 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:29:41.940967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls podName:9e5c729b-f3da-4888-a687-0d21c3afea6d nodeName:}" failed. No retries permitted until 2026-07-27 21:29:42.440952902 +0000 UTC m=+1511.105578124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 27 21:29:41.941311 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.940896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.941311 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.941209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.941423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.941312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.941479 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.941447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.941525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.941497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.941647 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.941630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:41.949599 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.949580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqnx\" (UniqueName: \"kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:41.949708 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:41.949633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfj4\" (UniqueName: \"kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:42.018790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.018763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f" event={"ID":"cccc1a46-1cf6-4a07-b505-02fb14224a61","Type":"ContainerDied","Data":"8d7b73572e95277550738e93de27e773393c8f3b40cea74e96d1f088ce449e4d"}
Jul 27 21:29:42.018790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.018777    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"
Jul 27 21:29:42.019315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.018824    2580 scope.go:117] "RemoveContainer" containerID="b2cf931c3bb139b47d9efbc4de138da7b6721860d984b5994699fa2c8e611d8c"
Jul 27 21:29:42.027167 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.027158    2580 scope.go:117] "RemoveContainer" containerID="007fa55e3db155c1ec8f20af97fb35f9d78e4aa423ef4f9ff572494868ca5dc8"
Jul 27 21:29:42.036333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.036310    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:29:42.038122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.038090    2580 scope.go:117] "RemoveContainer" containerID="6edd1fac87caee978cd6fd38b1150d74b7dfcf67b9429516f42285a3e73c7251"
Jul 27 21:29:42.041349 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.041322    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f"]
Jul 27 21:29:42.443820 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.443794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:42.443994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.443834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:42.448285 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.447547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:42.448285 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.447549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:42.686013 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.685980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:42.695622 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.695568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:42.833635 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.833611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:29:42.836040 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:29:42.836012    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5c729b_f3da_4888_a687_0d21c3afea6d.slice/crio-4bc68c8fee233bb9e2cd51e07d71f2be0fa5d138dfd5d707a20c31820ac6ac4d WatchSource:0}: Error finding container 4bc68c8fee233bb9e2cd51e07d71f2be0fa5d138dfd5d707a20c31820ac6ac4d: Status 404 returned error can't find the container with id 4bc68c8fee233bb9e2cd51e07d71f2be0fa5d138dfd5d707a20c31820ac6ac4d
Jul 27 21:29:42.850722 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:42.850695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:29:42.852338 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:29:42.852318    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991b44d7_0c58_4bbc_9619_397862744783.slice/crio-7e2eebf7f7a1e67736b292d5ec82385c4168ebbce9ec8a22df3017e18b359910 WatchSource:0}: Error finding container 7e2eebf7f7a1e67736b292d5ec82385c4168ebbce9ec8a22df3017e18b359910: Status 404 returned error can't find the container with id 7e2eebf7f7a1e67736b292d5ec82385c4168ebbce9ec8a22df3017e18b359910
Jul 27 21:29:43.025303 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:43.025193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerStarted","Data":"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"}
Jul 27 21:29:43.025303 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:43.025297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerStarted","Data":"4bc68c8fee233bb9e2cd51e07d71f2be0fa5d138dfd5d707a20c31820ac6ac4d"}
Jul 27 21:29:43.027833 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:43.027797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerStarted","Data":"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"}
Jul 27 21:29:43.027969 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:43.027856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerStarted","Data":"7e2eebf7f7a1e67736b292d5ec82385c4168ebbce9ec8a22df3017e18b359910"}
Jul 27 21:29:43.878273 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:43.878231    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccc1a46-1cf6-4a07-b505-02fb14224a61" path="/var/lib/kubelet/pods/cccc1a46-1cf6-4a07-b505-02fb14224a61/volumes"
Jul 27 21:29:47.046519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:47.046433    2580 generic.go:356] "Generic (PLEG): container finished" podID="991b44d7-0c58-4bbc-9619-397862744783" containerID="59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b" exitCode=0
Jul 27 21:29:47.046519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:47.046505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerDied","Data":"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"}
Jul 27 21:29:47.048079 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:47.048060    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerID="9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980" exitCode=0
Jul 27 21:29:47.048156 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:47.048088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerDied","Data":"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"}
Jul 27 21:29:48.054671 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.054634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerStarted","Data":"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"}
Jul 27 21:29:48.055122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.054709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerStarted","Data":"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"}
Jul 27 21:29:48.055122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.055026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:48.055282 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.055133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:48.056657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.056629    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:29:48.056774 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.056763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerStarted","Data":"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"}
Jul 27 21:29:48.056833 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.056801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerStarted","Data":"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"}
Jul 27 21:29:48.056995 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.056980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:48.073199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.073146    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podStartSLOduration=3.231306437 podStartE2EDuration="7.073134632s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841828195s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:48.072646851 +0000 UTC m=+1516.737272088" watchObservedRunningTime="2026-07-27 21:29:48.073134632 +0000 UTC m=+1516.737759940"
Jul 27 21:29:48.089769 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:48.089728    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podStartSLOduration=3.490791623 podStartE2EDuration="7.089717389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.598925766s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:48.08901847 +0000 UTC m=+1516.753643709" watchObservedRunningTime="2026-07-27 21:29:48.089717389 +0000 UTC m=+1516.754342627"
Jul 27 21:29:49.062009 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:49.061977    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:29:49.062487 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:49.062154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:49.063209 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:49.063185    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:29:50.067694 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:50.067651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:29:54.067206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:54.067129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:29:54.067637 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:54.067611    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:29:55.073332 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:55.073302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:29:55.073864 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:29:55.073835    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:04.068433 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:04.068386    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:05.073956 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:05.073917    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:14.068112 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:14.068071    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:15.074749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:15.074711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:24.068384 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:24.068343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:25.073834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:25.073796    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:34.067864 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:34.067824    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:35.074661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:35.074622    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:44.068071 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:44.068019    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:45.074357 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:45.074319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:30:54.068081 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:54.068033    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:30:55.073910 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:30:55.073874    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:31:04.068398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:04.068369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:31:05.074493 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:05.074466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:31:11.839559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:11.839528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:31:11.839952 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:11.839755    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" containerID="cri-o://71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4" gracePeriod=30
Jul 27 21:31:11.839952 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:11.839799    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kube-rbac-proxy" containerID="cri-o://fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e" gracePeriod=30
Jul 27 21:31:12.445882 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:12.445848    2580 generic.go:356] "Generic (PLEG): container finished" podID="991b44d7-0c58-4bbc-9619-397862744783" containerID="fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e" exitCode=2
Jul 27 21:31:12.446060 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:12.445927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerDied","Data":"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"}
Jul 27 21:31:15.068422 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:15.068381    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 27 21:31:15.074737 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:15.074711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 21:31:16.387648 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.387627    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:31:16.467872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.467803    2580 generic.go:356] "Generic (PLEG): container finished" podID="991b44d7-0c58-4bbc-9619-397862744783" containerID="71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4" exitCode=0
Jul 27 21:31:16.467872 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.467864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"
Jul 27 21:31:16.468034 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.467878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerDied","Data":"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"}
Jul 27 21:31:16.468034 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.467937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs" event={"ID":"991b44d7-0c58-4bbc-9619-397862744783","Type":"ContainerDied","Data":"7e2eebf7f7a1e67736b292d5ec82385c4168ebbce9ec8a22df3017e18b359910"}
Jul 27 21:31:16.468034 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.467963    2580 scope.go:117] "RemoveContainer" containerID="fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"
Jul 27 21:31:16.476379 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.476360    2580 scope.go:117] "RemoveContainer" containerID="71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"
Jul 27 21:31:16.483887 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.483871    2580 scope.go:117] "RemoveContainer" containerID="59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"
Jul 27 21:31:16.491068 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491035    2580 scope.go:117] "RemoveContainer" containerID="fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"
Jul 27 21:31:16.491323 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:31:16.491304    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e\": container with ID starting with fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e not found: ID does not exist" containerID="fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"
Jul 27 21:31:16.491397 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491335    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e"} err="failed to get container status \"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e\": rpc error: code = NotFound desc = could not find container \"fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e\": container with ID starting with fb8cd6e181d9c4e5fc7bf48a030040e502f06ba650606ebf99ecd4209c1bc09e not found: ID does not exist"
Jul 27 21:31:16.491397 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491357    2580 scope.go:117] "RemoveContainer" containerID="71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"
Jul 27 21:31:16.491597 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:31:16.491581    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4\": container with ID starting with 71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4 not found: ID does not exist" containerID="71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"
Jul 27 21:31:16.491643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491604    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4"} err="failed to get container status \"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4\": rpc error: code = NotFound desc = could not find container \"71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4\": container with ID starting with 71c469297b0f5f8e2ff6de88762bf2ecfc58bd5d1aa5f0ceeffc895f8fbb56c4 not found: ID does not exist"
Jul 27 21:31:16.491643 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491618    2580 scope.go:117] "RemoveContainer" containerID="59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"
Jul 27 21:31:16.491908 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:31:16.491881    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b\": container with ID starting with 59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b not found: ID does not exist" containerID="59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"
Jul 27 21:31:16.491965 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.491916    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b"} err="failed to get container status \"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b\": rpc error: code = NotFound desc = could not find container \"59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b\": container with ID starting with 59cdd43b7b005002d5782bfc0015c1c33da58d3b4535e24b72edb0881b1c9f1b not found: ID does not exist"
Jul 27 21:31:16.524201 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert\") pod \"991b44d7-0c58-4bbc-9619-397862744783\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") "
Jul 27 21:31:16.524331 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") pod \"991b44d7-0c58-4bbc-9619-397862744783\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") "
Jul 27 21:31:16.524398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config\") pod \"991b44d7-0c58-4bbc-9619-397862744783\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") "
Jul 27 21:31:16.524398 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqnx\" (UniqueName: \"kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx\") pod \"991b44d7-0c58-4bbc-9619-397862744783\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") "
Jul 27 21:31:16.524502 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location\") pod \"991b44d7-0c58-4bbc-9619-397862744783\" (UID: \"991b44d7-0c58-4bbc-9619-397862744783\") "
Jul 27 21:31:16.524502 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "991b44d7-0c58-4bbc-9619-397862744783" (UID: "991b44d7-0c58-4bbc-9619-397862744783"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:16.524666 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524651    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:31:16.524745 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "991b44d7-0c58-4bbc-9619-397862744783" (UID: "991b44d7-0c58-4bbc-9619-397862744783"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:16.524785 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.524742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "991b44d7-0c58-4bbc-9619-397862744783" (UID: "991b44d7-0c58-4bbc-9619-397862744783"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:16.526250 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.526230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "991b44d7-0c58-4bbc-9619-397862744783" (UID: "991b44d7-0c58-4bbc-9619-397862744783"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:31:16.526454 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.526435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx" (OuterVolumeSpecName: "kube-api-access-4rqnx") pod "991b44d7-0c58-4bbc-9619-397862744783" (UID: "991b44d7-0c58-4bbc-9619-397862744783"). InnerVolumeSpecName "kube-api-access-4rqnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:31:16.625310 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.625275    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/991b44d7-0c58-4bbc-9619-397862744783-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:31:16.625310 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.625307    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/991b44d7-0c58-4bbc-9619-397862744783-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:31:16.625310 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.625318    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqnx\" (UniqueName: \"kubernetes.io/projected/991b44d7-0c58-4bbc-9619-397862744783-kube-api-access-4rqnx\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:31:16.625559 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.625327    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/991b44d7-0c58-4bbc-9619-397862744783-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:31:16.790517 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.790481    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:31:16.794405 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:16.794384    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs"]
Jul 27 21:31:17.877632 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:31:17.877600    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991b44d7-0c58-4bbc-9619-397862744783" path="/var/lib/kubelet/pods/991b44d7-0c58-4bbc-9619-397862744783/volumes"
Jul 27 21:32:31.954727 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:31.954689    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:32:31.955162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:31.954982    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" containerID="cri-o://2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c" gracePeriod=30
Jul 27 21:32:31.955162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:31.955023    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kube-rbac-proxy" containerID="cri-o://6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15" gracePeriod=30
Jul 27 21:32:32.807142 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:32.807109    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerID="6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15" exitCode=2
Jul 27 21:32:32.807346 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:32.807158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerDied","Data":"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"}
Jul 27 21:32:34.062076 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:34.062029    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 27 21:32:34.068521 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:34.068486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:32:36.606917 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.606893    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:32:36.679953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.679891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location\") pod \"9e5c729b-f3da-4888-a687-0d21c3afea6d\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") "
Jul 27 21:32:36.679953 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.679927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxfj4\" (UniqueName: \"kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4\") pod \"9e5c729b-f3da-4888-a687-0d21c3afea6d\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") "
Jul 27 21:32:36.680135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.679983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config\") pod \"9e5c729b-f3da-4888-a687-0d21c3afea6d\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") "
Jul 27 21:32:36.680135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.680006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert\") pod \"9e5c729b-f3da-4888-a687-0d21c3afea6d\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") "
Jul 27 21:32:36.680135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.680046    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") pod \"9e5c729b-f3da-4888-a687-0d21c3afea6d\" (UID: \"9e5c729b-f3da-4888-a687-0d21c3afea6d\") "
Jul 27 21:32:36.680328 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.680301    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e5c729b-f3da-4888-a687-0d21c3afea6d" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:32:36.680423 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.680397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e5c729b-f3da-4888-a687-0d21c3afea6d" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:32:36.680472 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.680402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e5c729b-f3da-4888-a687-0d21c3afea6d" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:32:36.682156 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.682134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e5c729b-f3da-4888-a687-0d21c3afea6d" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:32:36.682156 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.682141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4" (OuterVolumeSpecName: "kube-api-access-cxfj4") pod "9e5c729b-f3da-4888-a687-0d21c3afea6d" (UID: "9e5c729b-f3da-4888-a687-0d21c3afea6d"). InnerVolumeSpecName "kube-api-access-cxfj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:32:36.781476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.781453    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:32:36.781476 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.781472    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c729b-f3da-4888-a687-0d21c3afea6d-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:32:36.781614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.781483    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e5c729b-f3da-4888-a687-0d21c3afea6d-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:32:36.781614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.781492    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e5c729b-f3da-4888-a687-0d21c3afea6d-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:32:36.781614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.781501    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfj4\" (UniqueName: \"kubernetes.io/projected/9e5c729b-f3da-4888-a687-0d21c3afea6d-kube-api-access-cxfj4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:32:36.832949 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.832918    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerID="2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c" exitCode=0
Jul 27 21:32:36.833102 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.832961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerDied","Data":"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"}
Jul 27 21:32:36.833102 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.832970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"
Jul 27 21:32:36.833102 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.833017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn" event={"ID":"9e5c729b-f3da-4888-a687-0d21c3afea6d","Type":"ContainerDied","Data":"4bc68c8fee233bb9e2cd51e07d71f2be0fa5d138dfd5d707a20c31820ac6ac4d"}
Jul 27 21:32:36.833102 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.833043    2580 scope.go:117] "RemoveContainer" containerID="6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"
Jul 27 21:32:36.841920 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.841902    2580 scope.go:117] "RemoveContainer" containerID="2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"
Jul 27 21:32:36.849615 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.849598    2580 scope.go:117] "RemoveContainer" containerID="9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"
Jul 27 21:32:36.852525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.852506    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:32:36.855718 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.855697    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn"]
Jul 27 21:32:36.858127 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858112    2580 scope.go:117] "RemoveContainer" containerID="6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"
Jul 27 21:32:36.858393 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:36.858372    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15\": container with ID starting with 6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15 not found: ID does not exist" containerID="6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"
Jul 27 21:32:36.858484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858403    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15"} err="failed to get container status \"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15\": rpc error: code = NotFound desc = could not find container \"6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15\": container with ID starting with 6ab876cf1024d2f1ea892ab69b779f759ec0d7ec0ca93720b52db2158ac3dd15 not found: ID does not exist"
Jul 27 21:32:36.858484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858422    2580 scope.go:117] "RemoveContainer" containerID="2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"
Jul 27 21:32:36.858684 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:36.858667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c\": container with ID starting with 2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c not found: ID does not exist" containerID="2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"
Jul 27 21:32:36.858730 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858691    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c"} err="failed to get container status \"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c\": rpc error: code = NotFound desc = could not find container \"2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c\": container with ID starting with 2862aa6ff7fd5fea7783d7dfc1418732d45f61e52836f27c1a615fbcf887a20c not found: ID does not exist"
Jul 27 21:32:36.858730 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858706    2580 scope.go:117] "RemoveContainer" containerID="9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"
Jul 27 21:32:36.858953 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:36.858937    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980\": container with ID starting with 9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980 not found: ID does not exist" containerID="9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"
Jul 27 21:32:36.859001 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:36.858957    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980"} err="failed to get container status \"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980\": rpc error: code = NotFound desc = could not find container \"9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980\": container with ID starting with 9e46db2ce3153f5f78067cfb0a61f8c1309ae8ca42c564aabe2189357ff41980 not found: ID does not exist"
Jul 27 21:32:37.132978 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.132942    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:32:37.133300 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="storage-initializer"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133302    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="storage-initializer"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133315    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133322    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133329    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133334    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133343    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container"
Jul 27 21:32:37.133348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133348    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133361    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="storage-initializer"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133366    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="storage-initializer"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133376    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133380    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133436    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kube-rbac-proxy"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133444    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="991b44d7-0c58-4bbc-9619-397862744783" containerName="kserve-container"
Jul 27 21:32:37.133553 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.133450    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" containerName="kserve-container"
Jul 27 21:32:37.138041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.138020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.140477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:32:37.140477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 27 21:32:37.140649 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:32:37.140744 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:32:37.140809 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140741    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:32:37.140809 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:32:37.140809 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.140732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 27 21:32:37.147456 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.147436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:32:37.149048 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.149031    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:32:37.156451 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.156433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.159002 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.158985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 27 21:32:37.160206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.160185    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:32:37.184928 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.184903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.185052 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.184935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.185052 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.184956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.185052 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.184973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.185160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbhq\" (UniqueName: \"kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.185160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.185160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.185160 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.185318 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.185318 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.185216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnn7\" (UniqueName: \"kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286058 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286058 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbhq\" (UniqueName: \"kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286545 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286545 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286545 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnn7\" (UniqueName: \"kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286545 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286750 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:37.286632    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 21:32:37.286750 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286750 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286750 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:37.286699    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls podName:e6227be4-d2ae-4755-a0a4-f907c5657abb nodeName:}" failed. No retries permitted until 2026-07-27 21:32:37.78668023 +0000 UTC m=+1686.451305463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-jb7wd" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 21:32:37.286972 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:37.286750    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 27 21:32:37.286972 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:32:37.286800    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls podName:34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a nodeName:}" failed. No retries permitted until 2026-07-27 21:32:37.786783808 +0000 UTC m=+1686.451409024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls") pod "isvc-canary-stop-v2-predictor-77f778d864-5hxnt" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 27 21:32:37.286972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.286972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.286972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.286950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.287149 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.287064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.294883 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.294859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnn7\" (UniqueName: \"kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.295025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.294936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbhq\" (UniqueName: \"kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.790347 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.790302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.790807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.790355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.792919 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.792892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-jb7wd\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:37.793050 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.792892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-5hxnt\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:37.883385 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:37.883353    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5c729b-f3da-4888-a687-0d21c3afea6d" path="/var/lib/kubelet/pods/9e5c729b-f3da-4888-a687-0d21c3afea6d/volumes"
Jul 27 21:32:38.049876 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.049798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:38.068143 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.068118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:38.198156 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.198113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:32:38.200887 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:32:38.200854    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6227be4_d2ae_4755_a0a4_f907c5657abb.slice/crio-160dc3fad59e73b42d274b03f335ad7b2976011bd01d3a0dc6aec6bf1a8d4184 WatchSource:0}: Error finding container 160dc3fad59e73b42d274b03f335ad7b2976011bd01d3a0dc6aec6bf1a8d4184: Status 404 returned error can't find the container with id 160dc3fad59e73b42d274b03f335ad7b2976011bd01d3a0dc6aec6bf1a8d4184
Jul 27 21:32:38.221678 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.221658    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:32:38.233309 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:32:38.233281    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fe4bcd_039e_4aff_9f52_b8f9dc14fe7a.slice/crio-f636d55460071394077f75220076f54709d1a5fad2f7a7c718ddf94ffd739c0d WatchSource:0}: Error finding container f636d55460071394077f75220076f54709d1a5fad2f7a7c718ddf94ffd739c0d: Status 404 returned error can't find the container with id f636d55460071394077f75220076f54709d1a5fad2f7a7c718ddf94ffd739c0d
Jul 27 21:32:38.846864 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.846832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerStarted","Data":"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"}
Jul 27 21:32:38.847313 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.846892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerStarted","Data":"160dc3fad59e73b42d274b03f335ad7b2976011bd01d3a0dc6aec6bf1a8d4184"}
Jul 27 21:32:38.848446 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.848414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerStarted","Data":"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"}
Jul 27 21:32:38.848573 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:38.848477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerStarted","Data":"f636d55460071394077f75220076f54709d1a5fad2f7a7c718ddf94ffd739c0d"}
Jul 27 21:32:42.867811 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:42.867777    2580 generic.go:356] "Generic (PLEG): container finished" podID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerID="a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a" exitCode=0
Jul 27 21:32:42.868279 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:42.867852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerDied","Data":"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"}
Jul 27 21:32:42.869537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:42.869517    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerID="2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff" exitCode=0
Jul 27 21:32:42.869621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:42.869574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerDied","Data":"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"}
Jul 27 21:32:43.880317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.880273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerStarted","Data":"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"}
Jul 27 21:32:43.880777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.880347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerStarted","Data":"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"}
Jul 27 21:32:43.880777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.880546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:43.881616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.881599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerStarted","Data":"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"}
Jul 27 21:32:43.881675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.881638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerStarted","Data":"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"}
Jul 27 21:32:43.881824 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.881805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:43.881869 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.881859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:43.883157 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.883127    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:32:43.909357 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.909313    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podStartSLOduration=3.134614655 podStartE2EDuration="6.909299695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77468504s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:32:43.904823295 +0000 UTC m=+1692.569448533" watchObservedRunningTime="2026-07-27 21:32:43.909299695 +0000 UTC m=+1692.573924932"
Jul 27 21:32:43.936790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:43.936749    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podStartSLOduration=3.140342154 podStartE2EDuration="6.936734912s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796392758s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:32:43.934303163 +0000 UTC m=+1692.598928425" watchObservedRunningTime="2026-07-27 21:32:43.936734912 +0000 UTC m=+1692.601360149"
Jul 27 21:32:44.886751 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:44.886715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:44.886751 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:44.886740    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:32:44.888031 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:44.888006    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:32:45.890754 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:45.890719    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:32:49.894445 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:49.894411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:32:49.894906 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:49.894834    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:32:50.895610 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:50.895582    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:32:50.896252 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:50.896228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:32:59.895519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:32:59.895441    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:00.896805 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:00.896771    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:09.895481 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:09.895440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:10.897078 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:10.897040    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:19.895086 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:19.895049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:20.896780 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:20.896740    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:29.895873 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:29.895837    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:30.896753 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:30.896713    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:39.895531 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:39.895491    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:40.896154 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:40.896115    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:49.895101 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:49.895066    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:33:50.897164 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:50.897126    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:33:58.873396 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:58.873368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:33:59.895370 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:33:59.895345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:34:07.166284 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.166237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:34:07.166704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.166479    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" containerID="cri-o://daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f" gracePeriod=30
Jul 27 21:34:07.166704 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.166529    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kube-rbac-proxy" containerID="cri-o://c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632" gracePeriod=30
Jul 27 21:34:07.202772 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.202738    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:34:07.203024 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.202979    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" containerID="cri-o://2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c" gracePeriod=30
Jul 27 21:34:07.203129 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:07.203004    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kube-rbac-proxy" containerID="cri-o://1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98" gracePeriod=30
Jul 27 21:34:08.270298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:08.270243    2580 generic.go:356] "Generic (PLEG): container finished" podID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerID="1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98" exitCode=2
Jul 27 21:34:08.270298 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:08.270288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerDied","Data":"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"}
Jul 27 21:34:08.272214 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:08.272194    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerID="c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632" exitCode=2
Jul 27 21:34:08.272358 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:08.272228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerDied","Data":"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"}
Jul 27 21:34:08.873009 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:08.872970    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:34:09.887679 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:09.887648    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 27 21:34:09.895593 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:09.895570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:34:10.890991 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:10.890944    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 27 21:34:11.637115 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.637093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:34:11.697860 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.697832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location\") pod \"e6227be4-d2ae-4755-a0a4-f907c5657abb\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") "
Jul 27 21:34:11.698004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.697900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config\") pod \"e6227be4-d2ae-4755-a0a4-f907c5657abb\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") "
Jul 27 21:34:11.698004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.697963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfnn7\" (UniqueName: \"kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7\") pod \"e6227be4-d2ae-4755-a0a4-f907c5657abb\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") "
Jul 27 21:34:11.698004 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.697993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") pod \"e6227be4-d2ae-4755-a0a4-f907c5657abb\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") "
Jul 27 21:34:11.698190 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.698081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert\") pod \"e6227be4-d2ae-4755-a0a4-f907c5657abb\" (UID: \"e6227be4-d2ae-4755-a0a4-f907c5657abb\") "
Jul 27 21:34:11.698475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.698444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6227be4-d2ae-4755-a0a4-f907c5657abb" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:11.698947 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.698899    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6227be4-d2ae-4755-a0a4-f907c5657abb" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:11.699072 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.699028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6227be4-d2ae-4755-a0a4-f907c5657abb" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:34:11.701020 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.700994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6227be4-d2ae-4755-a0a4-f907c5657abb" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:34:11.701169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.701150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7" (OuterVolumeSpecName: "kube-api-access-tfnn7") pod "e6227be4-d2ae-4755-a0a4-f907c5657abb" (UID: "e6227be4-d2ae-4755-a0a4-f907c5657abb"). InnerVolumeSpecName "kube-api-access-tfnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:34:11.752940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.752920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:34:11.799513 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config\") pod \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") "
Jul 27 21:34:11.799646 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjbhq\" (UniqueName: \"kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq\") pod \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") "
Jul 27 21:34:11.799646 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") pod \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") "
Jul 27 21:34:11.799646 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert\") pod \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") "
Jul 27 21:34:11.799822 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location\") pod \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\" (UID: \"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a\") "
Jul 27 21:34:11.799880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:11.800021 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.799999    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800021 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800020    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnn7\" (UniqueName: \"kubernetes.io/projected/e6227be4-d2ae-4755-a0a4-f907c5657abb-kube-api-access-tfnn7\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800029    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6227be4-d2ae-4755-a0a4-f907c5657abb-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800038    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6227be4-d2ae-4755-a0a4-f907c5657abb-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800047    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800055    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6227be4-d2ae-4755-a0a4-f907c5657abb-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:11.800162 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.800070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:34:11.801719 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.801693    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq" (OuterVolumeSpecName: "kube-api-access-xjbhq") pod "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a"). InnerVolumeSpecName "kube-api-access-xjbhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:34:11.801719 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.801700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" (UID: "34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:34:11.900824 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.900799    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.901116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.900840    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbhq\" (UniqueName: \"kubernetes.io/projected/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-kube-api-access-xjbhq\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.901116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.900850    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:11.901116 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:11.900859    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:34:12.291697 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.291659    2580 generic.go:356] "Generic (PLEG): container finished" podID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerID="2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c" exitCode=0
Jul 27 21:34:12.291877 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.291714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"
Jul 27 21:34:12.291877 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.291731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerDied","Data":"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"}
Jul 27 21:34:12.291877 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.291800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt" event={"ID":"34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a","Type":"ContainerDied","Data":"f636d55460071394077f75220076f54709d1a5fad2f7a7c718ddf94ffd739c0d"}
Jul 27 21:34:12.292535 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.291840    2580 scope.go:117] "RemoveContainer" containerID="1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"
Jul 27 21:34:12.293768 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.293726    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerID="daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f" exitCode=0
Jul 27 21:34:12.293768 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.293761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerDied","Data":"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"}
Jul 27 21:34:12.293929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.293787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"
Jul 27 21:34:12.293929 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.293809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd" event={"ID":"e6227be4-d2ae-4755-a0a4-f907c5657abb","Type":"ContainerDied","Data":"160dc3fad59e73b42d274b03f335ad7b2976011bd01d3a0dc6aec6bf1a8d4184"}
Jul 27 21:34:12.301112 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.301098    2580 scope.go:117] "RemoveContainer" containerID="2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"
Jul 27 21:34:12.310188 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.310150    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:34:12.312043 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.312023    2580 scope.go:117] "RemoveContainer" containerID="a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"
Jul 27 21:34:12.314101 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.314079    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd"]
Jul 27 21:34:12.319618 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.319604    2580 scope.go:117] "RemoveContainer" containerID="1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"
Jul 27 21:34:12.319873 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.319853    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98\": container with ID starting with 1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98 not found: ID does not exist" containerID="1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"
Jul 27 21:34:12.319940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.319884    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98"} err="failed to get container status \"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98\": rpc error: code = NotFound desc = could not find container \"1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98\": container with ID starting with 1f229877d2b60dd2768fb2def47443830e1ef77156bcd68f1c8bf81cca5d9a98 not found: ID does not exist"
Jul 27 21:34:12.319940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.319905    2580 scope.go:117] "RemoveContainer" containerID="2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"
Jul 27 21:34:12.320173 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.320156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c\": container with ID starting with 2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c not found: ID does not exist" containerID="2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"
Jul 27 21:34:12.320211 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.320179    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c"} err="failed to get container status \"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c\": rpc error: code = NotFound desc = could not find container \"2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c\": container with ID starting with 2e410f72385dfdef726878b9a12ed052062185187fcc6f769bf2fceb4d1f5d3c not found: ID does not exist"
Jul 27 21:34:12.320211 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.320199    2580 scope.go:117] "RemoveContainer" containerID="a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"
Jul 27 21:34:12.320427 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.320412    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a\": container with ID starting with a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a not found: ID does not exist" containerID="a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"
Jul 27 21:34:12.320475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.320432    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a"} err="failed to get container status \"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a\": rpc error: code = NotFound desc = could not find container \"a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a\": container with ID starting with a44f7c1143b88e2b71bf195ce72e4a89ebf56c3dbfd500fece68a6e975eb5f9a not found: ID does not exist"
Jul 27 21:34:12.320475 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.320445    2580 scope.go:117] "RemoveContainer" containerID="c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"
Jul 27 21:34:12.324095 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.324077    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:34:12.329200 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.329179    2580 scope.go:117] "RemoveContainer" containerID="daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"
Jul 27 21:34:12.329392 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.329372    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt"]
Jul 27 21:34:12.336385 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.336370    2580 scope.go:117] "RemoveContainer" containerID="2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"
Jul 27 21:34:12.343325 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.343310    2580 scope.go:117] "RemoveContainer" containerID="c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"
Jul 27 21:34:12.343562 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.343542    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632\": container with ID starting with c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632 not found: ID does not exist" containerID="c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"
Jul 27 21:34:12.343634 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.343570    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632"} err="failed to get container status \"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632\": rpc error: code = NotFound desc = could not find container \"c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632\": container with ID starting with c3348d603b06fad29bfb5a886df2a9860e77d9de0e8b5bc34924714945d8b632 not found: ID does not exist"
Jul 27 21:34:12.343634 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.343592    2580 scope.go:117] "RemoveContainer" containerID="daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"
Jul 27 21:34:12.343839 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.343823    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f\": container with ID starting with daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f not found: ID does not exist" containerID="daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"
Jul 27 21:34:12.343880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.343848    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f"} err="failed to get container status \"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f\": rpc error: code = NotFound desc = could not find container \"daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f\": container with ID starting with daee6214b513e63979d5adde4415868cf52b923363c2d29e8d92cab3c32f512f not found: ID does not exist"
Jul 27 21:34:12.343880 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.343865    2580 scope.go:117] "RemoveContainer" containerID="2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"
Jul 27 21:34:12.344069 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:12.344053    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff\": container with ID starting with 2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff not found: ID does not exist" containerID="2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"
Jul 27 21:34:12.344129 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:12.344075    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff"} err="failed to get container status \"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff\": rpc error: code = NotFound desc = could not find container \"2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff\": container with ID starting with 2d2287d719c0c59909a1e91dd80fb728c629b6cd26a94bcad853b839c3d626ff not found: ID does not exist"
Jul 27 21:34:13.878342 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:13.878310    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" path="/var/lib/kubelet/pods/34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a/volumes"
Jul 27 21:34:13.878908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:13.878890    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" path="/var/lib/kubelet/pods/e6227be4-d2ae-4755-a0a4-f907c5657abb/volumes"
Jul 27 21:34:17.469822 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.469791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="storage-initializer"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470155    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="storage-initializer"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470164    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="storage-initializer"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470169    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="storage-initializer"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470178    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470183    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470195    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470202    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470207    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470220    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container"
Jul 27 21:34:17.470222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470225    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container"
Jul 27 21:34:17.470701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470289    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kserve-container"
Jul 27 21:34:17.470701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470299    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6227be4-d2ae-4755-a0a4-f907c5657abb" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470305    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kube-rbac-proxy"
Jul 27 21:34:17.470701 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.470310    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fe4bcd-039e-4aff-9f52-b8f9dc14fe7a" containerName="kserve-container"
Jul 27 21:34:17.475190 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.475173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.477889 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.477870    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:34:17.477997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.477892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:34:17.477997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.477931    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:34:17.477997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.477952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-56d0f5-predictor-serving-cert\""
Jul 27 21:34:17.477997 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.477969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:34:17.478165 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.478001    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:34:17.478764 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.478747    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-56d0f5-kube-rbac-proxy-sar-config\""
Jul 27 21:34:17.482760 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.482740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:34:17.547457 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.547435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.547581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.547470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.547581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.547494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nxk\" (UniqueName: \"kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.547581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.547516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.547717 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.547583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.648831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.648805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.648951 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.648834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nxk\" (UniqueName: \"kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.648951 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.648852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.649074 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.649004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.649152 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.649126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.649204 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:17.649178    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-serving-cert: secret "isvc-primary-56d0f5-predictor-serving-cert" not found
Jul 27 21:34:17.649272 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:34:17.649242    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls podName:55e9939e-e834-4cfc-8dec-705860a97578 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:18.149223221 +0000 UTC m=+1786.813848438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls") pod "isvc-primary-56d0f5-predictor-fb478888f-prb6c" (UID: "55e9939e-e834-4cfc-8dec-705860a97578") : secret "isvc-primary-56d0f5-predictor-serving-cert" not found
Jul 27 21:34:17.649333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.649273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.649526 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.649510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.649639 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.649624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:17.657245 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:17.657228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nxk\" (UniqueName: \"kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:18.153169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:18.153136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:18.155657 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:18.155637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") pod \"isvc-primary-56d0f5-predictor-fb478888f-prb6c\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") " pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:18.386194 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:18.386163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:18.517135 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:18.517109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:34:18.519568 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:34:18.519538    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e9939e_e834_4cfc_8dec_705860a97578.slice/crio-27e440a06050c0199926fca7ddd37df2b85c742d6c435fda9568be23ca7c8d3f WatchSource:0}: Error finding container 27e440a06050c0199926fca7ddd37df2b85c742d6c435fda9568be23ca7c8d3f: Status 404 returned error can't find the container with id 27e440a06050c0199926fca7ddd37df2b85c742d6c435fda9568be23ca7c8d3f
Jul 27 21:34:19.326940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:19.326904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerStarted","Data":"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"}
Jul 27 21:34:19.327089 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:19.326959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerStarted","Data":"27e440a06050c0199926fca7ddd37df2b85c742d6c435fda9568be23ca7c8d3f"}
Jul 27 21:34:22.342231 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:22.342201    2580 generic.go:356] "Generic (PLEG): container finished" podID="55e9939e-e834-4cfc-8dec-705860a97578" containerID="c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a" exitCode=0
Jul 27 21:34:22.342597 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:22.342246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerDied","Data":"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"}
Jul 27 21:34:23.349368 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:23.349329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerStarted","Data":"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"}
Jul 27 21:34:23.349846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:23.349403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerStarted","Data":"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"}
Jul 27 21:34:23.349846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:23.349626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:23.369722 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:23.369675    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podStartSLOduration=2.7371153599999998 podStartE2EDuration="6.369660799s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632545439s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:23.366877432 +0000 UTC m=+1792.031502675" watchObservedRunningTime="2026-07-27 21:34:23.369660799 +0000 UTC m=+1792.034286037"
Jul 27 21:34:24.353663 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:24.353637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:24.354891 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:24.354866    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:34:25.358122 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:25.358078    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:34:30.362642 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:30.362612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:34:30.363199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:30.363164    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:34:40.363759 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:40.363713    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:34:50.363791 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:34:50.363750    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:35:00.363396 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:35:00.363349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:35:10.363145 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:35:10.363111    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:35:20.363203 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:35:20.363164    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:35:30.363419 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:35:30.363383    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:35:39.878110 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:35:39.878080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:36:02.742069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.741989    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:02.745519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.745504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.748065 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.748041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-56d0f5-predictor-serving-cert\""
Jul 27 21:36:02.748201 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.748072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-56d0f5\""
Jul 27 21:36:02.748201 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.748072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-56d0f5-dockercfg-7ch9k\""
Jul 27 21:36:02.748201 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.748053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-56d0f5-kube-rbac-proxy-sar-config\""
Jul 27 21:36:02.755948 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.755927    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:02.891363 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.891335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.891519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.891373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.891519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.891398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.891519 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.891435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.891627 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.891543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qb8\" (UniqueName: \"kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992578 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.992499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.992579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qb8\" (UniqueName: \"kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.992621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.992648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992726 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.992683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.992951 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:02.992748    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-serving-cert: secret "isvc-secondary-56d0f5-predictor-serving-cert" not found
Jul 27 21:36:02.992951 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:02.992836    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls podName:4bec48ac-c5ac-47ff-a02d-d32fc74da9f4 nodeName:}" failed. No retries permitted until 2026-07-27 21:36:03.492811472 +0000 UTC m=+1892.157436705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls") pod "isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4") : secret "isvc-secondary-56d0f5-predictor-serving-cert" not found
Jul 27 21:36:02.993108 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.993087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.993288 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.993251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:02.993372 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:02.993353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:03.001786 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.001757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qb8\" (UniqueName: \"kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:03.496795 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.496759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:03.499198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.499181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") pod \"isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") " pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:03.656838 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.656784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:03.790402 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.790378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:03.792431 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:36:03.792395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bec48ac_c5ac_47ff_a02d_d32fc74da9f4.slice/crio-94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d WatchSource:0}: Error finding container 94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d: Status 404 returned error can't find the container with id 94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d
Jul 27 21:36:03.809673 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:03.809643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerStarted","Data":"94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d"}
Jul 27 21:36:04.815754 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:04.815718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerStarted","Data":"74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"}
Jul 27 21:36:08.836456 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:08.836428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/0.log"
Jul 27 21:36:08.836859 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:08.836472    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerID="74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777" exitCode=1
Jul 27 21:36:08.836859 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:08.836547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerDied","Data":"74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"}
Jul 27 21:36:09.843118 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:09.843091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/0.log"
Jul 27 21:36:09.843546 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:09.843169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerStarted","Data":"f2a62f121fa7c0c083106258427d94c6b23161cf10ff6b695031e68b409cec6c"}
Jul 27 21:36:14.867408 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.867378    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/1.log"
Jul 27 21:36:14.867804 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.867750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/0.log"
Jul 27 21:36:14.867804 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.867781    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerID="f2a62f121fa7c0c083106258427d94c6b23161cf10ff6b695031e68b409cec6c" exitCode=1
Jul 27 21:36:14.867892 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.867853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerDied","Data":"f2a62f121fa7c0c083106258427d94c6b23161cf10ff6b695031e68b409cec6c"}
Jul 27 21:36:14.867934 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.867920    2580 scope.go:117] "RemoveContainer" containerID="74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"
Jul 27 21:36:14.868296 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.868248    2580 scope.go:117] "RemoveContainer" containerID="74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"
Jul 27 21:36:14.878905 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:14.878879    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_kserve-ci-e2e-test(4bec48ac-c5ac-47ff-a02d-d32fc74da9f4)\"" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"
Jul 27 21:36:14.879101 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:14.879075    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_kserve-ci-e2e-test_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4_0 in pod sandbox 94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d from index: no such id: '74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777'" containerID="74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"
Jul 27 21:36:14.879172 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:14.879110    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_kserve-ci-e2e-test_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4_0 in pod sandbox 94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d from index: no such id: '74372797545b791bf685a94200fa340704e7981d24e7b531ba0424f7440df777'"
Jul 27 21:36:15.873902 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:15.873876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/1.log"
Jul 27 21:36:20.696214 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.696172    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:36:20.696692 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.696520    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container" containerID="cri-o://09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d" gracePeriod=30
Jul 27 21:36:20.696692 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.696604    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kube-rbac-proxy" containerID="cri-o://2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31" gracePeriod=30
Jul 27 21:36:20.762267 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.762225    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:20.894414 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.894389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/1.log"
Jul 27 21:36:20.896836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.896813    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:20.897729 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.897709    2580 generic.go:356] "Generic (PLEG): container finished" podID="55e9939e-e834-4cfc-8dec-705860a97578" containerID="2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31" exitCode=2
Jul 27 21:36:20.897820 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.897779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerDied","Data":"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"}
Jul 27 21:36:20.898948 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.898933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/storage-initializer/1.log"
Jul 27 21:36:20.899037 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.898980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt" event={"ID":"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4","Type":"ContainerDied","Data":"94dc0d1626607583f7fda12de1ba0c92d11b569189879552c75d56cf446f327d"}
Jul 27 21:36:20.899037 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.899006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"
Jul 27 21:36:20.899037 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.899019    2580 scope.go:117] "RemoveContainer" containerID="f2a62f121fa7c0c083106258427d94c6b23161cf10ff6b695031e68b409cec6c"
Jul 27 21:36:20.949814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location\") pod \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") "
Jul 27 21:36:20.949814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config\") pod \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") "
Jul 27 21:36:20.950000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6qb8\" (UniqueName: \"kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8\") pod \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") "
Jul 27 21:36:20.950000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert\") pod \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") "
Jul 27 21:36:20.950000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") pod \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\" (UID: \"4bec48ac-c5ac-47ff-a02d-d32fc74da9f4\") "
Jul 27 21:36:20.950000 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.949969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:20.950194 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.950172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:20.950237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.950194    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:20.950237 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.950218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:20.951907 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.951887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8" (OuterVolumeSpecName: "kube-api-access-d6qb8") pod "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"). InnerVolumeSpecName "kube-api-access-d6qb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:20.952041 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:20.952024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" (UID: "4bec48ac-c5ac-47ff-a02d-d32fc74da9f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:21.050594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.050563    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:21.050594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.050589    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:21.050594 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.050598    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qb8\" (UniqueName: \"kubernetes.io/projected/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-kube-api-access-d6qb8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:21.050811 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.050609    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:21.234059 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.234032    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:21.238285 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.238242    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt"]
Jul 27 21:36:21.877812 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:21.877779    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" path="/var/lib/kubelet/pods/4bec48ac-c5ac-47ff-a02d-d32fc74da9f4/volumes"
Jul 27 21:36:24.853584 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.853561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:36:24.879457 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config\") pod \"55e9939e-e834-4cfc-8dec-705860a97578\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") "
Jul 27 21:36:24.879616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") pod \"55e9939e-e834-4cfc-8dec-705860a97578\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") "
Jul 27 21:36:24.879616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879545    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert\") pod \"55e9939e-e834-4cfc-8dec-705860a97578\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") "
Jul 27 21:36:24.879616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location\") pod \"55e9939e-e834-4cfc-8dec-705860a97578\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") "
Jul 27 21:36:24.879616 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4nxk\" (UniqueName: \"kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk\") pod \"55e9939e-e834-4cfc-8dec-705860a97578\" (UID: \"55e9939e-e834-4cfc-8dec-705860a97578\") "
Jul 27 21:36:24.879858 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55e9939e-e834-4cfc-8dec-705860a97578" (UID: "55e9939e-e834-4cfc-8dec-705860a97578"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:24.879915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55e9939e-e834-4cfc-8dec-705860a97578" (UID: "55e9939e-e834-4cfc-8dec-705860a97578"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:24.879915 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.879886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55e9939e-e834-4cfc-8dec-705860a97578" (UID: "55e9939e-e834-4cfc-8dec-705860a97578"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:24.881748 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.881729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55e9939e-e834-4cfc-8dec-705860a97578" (UID: "55e9939e-e834-4cfc-8dec-705860a97578"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:24.881848 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.881825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk" (OuterVolumeSpecName: "kube-api-access-h4nxk") pod "55e9939e-e834-4cfc-8dec-705860a97578" (UID: "55e9939e-e834-4cfc-8dec-705860a97578"). InnerVolumeSpecName "kube-api-access-h4nxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:24.924736 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.924648    2580 generic.go:356] "Generic (PLEG): container finished" podID="55e9939e-e834-4cfc-8dec-705860a97578" containerID="09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d" exitCode=0
Jul 27 21:36:24.924736 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.924703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerDied","Data":"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"}
Jul 27 21:36:24.924736 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.924722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"
Jul 27 21:36:24.924960 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.924792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c" event={"ID":"55e9939e-e834-4cfc-8dec-705860a97578","Type":"ContainerDied","Data":"27e440a06050c0199926fca7ddd37df2b85c742d6c435fda9568be23ca7c8d3f"}
Jul 27 21:36:24.924960 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.924829    2580 scope.go:117] "RemoveContainer" containerID="2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"
Jul 27 21:36:24.933763 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.933745    2580 scope.go:117] "RemoveContainer" containerID="09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"
Jul 27 21:36:24.941079 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.941063    2580 scope.go:117] "RemoveContainer" containerID="c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"
Jul 27 21:36:24.946600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.946579    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:36:24.949663 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.949642    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c"]
Jul 27 21:36:24.949928 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.949912    2580 scope.go:117] "RemoveContainer" containerID="2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"
Jul 27 21:36:24.950179 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:24.950156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31\": container with ID starting with 2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31 not found: ID does not exist" containerID="2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"
Jul 27 21:36:24.950230 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.950187    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31"} err="failed to get container status \"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31\": rpc error: code = NotFound desc = could not find container \"2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31\": container with ID starting with 2e7b673657e9c9f1598114b215fc5b07b5f1ff3d7b3e970ac9370114b81c0a31 not found: ID does not exist"
Jul 27 21:36:24.950230 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.950205    2580 scope.go:117] "RemoveContainer" containerID="09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"
Jul 27 21:36:24.950487 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:24.950469    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d\": container with ID starting with 09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d not found: ID does not exist" containerID="09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"
Jul 27 21:36:24.950540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.950492    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d"} err="failed to get container status \"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d\": rpc error: code = NotFound desc = could not find container \"09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d\": container with ID starting with 09cc59b81d08051def7c84b87512d5c9895f84e68a703d1221c48267e47b728d not found: ID does not exist"
Jul 27 21:36:24.950540 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.950507    2580 scope.go:117] "RemoveContainer" containerID="c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"
Jul 27 21:36:24.950720 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:24.950698    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a\": container with ID starting with c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a not found: ID does not exist" containerID="c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"
Jul 27 21:36:24.950822 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.950726    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a"} err="failed to get container status \"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a\": rpc error: code = NotFound desc = could not find container \"c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a\": container with ID starting with c1c3a3532de86f0f0ba501bd099fb2893f9a34e2fbeac0e6147a3d2becb8fa6a not found: ID does not exist"
Jul 27 21:36:24.980294 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.980249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nxk\" (UniqueName: \"kubernetes.io/projected/55e9939e-e834-4cfc-8dec-705860a97578-kube-api-access-h4nxk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:24.980294 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.980295    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:24.980422 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.980305    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55e9939e-e834-4cfc-8dec-705860a97578-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:24.980422 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.980313    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55e9939e-e834-4cfc-8dec-705860a97578-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:24.980422 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:24.980321    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55e9939e-e834-4cfc-8dec-705860a97578-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:25.880292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.880239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e9939e-e834-4cfc-8dec-705860a97578" path="/var/lib/kubelet/pods/55e9939e-e834-4cfc-8dec-705860a97578/volumes"
Jul 27 21:36:25.941234 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941200    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:25.941550 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941538    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941552    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kube-rbac-proxy"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941592    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kube-rbac-proxy"
Jul 27 21:36:25.941600 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941600    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="storage-initializer"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941605    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="storage-initializer"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941653    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941662    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kserve-container"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941668    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e9939e-e834-4cfc-8dec-705860a97578" containerName="kube-rbac-proxy"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941736    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.941777 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941741    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.941952 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.941788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bec48ac-c5ac-47ff-a02d-d32fc74da9f4" containerName="storage-initializer"
Jul 27 21:36:25.946238 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.946222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:25.948631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.948601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2f3164-dockercfg-7pzdt\""
Jul 27 21:36:25.948861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.948845    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:36:25.948948 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.948929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2f3164\""
Jul 27 21:36:25.949653 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.949632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2f3164-predictor-serving-cert\""
Jul 27 21:36:25.949756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.949696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:36:25.949756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.949732    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:36:25.949834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.949767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2f3164-kube-rbac-proxy-sar-config\""
Jul 27 21:36:25.954898 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.954877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:25.987996 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.987973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:25.988115 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.988015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:25.988115 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.988040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghxg\" (UniqueName: \"kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:25.988115 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.988094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:25.988284 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:25.988117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088735 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.088709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088896 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.088759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088896 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.088788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghxg\" (UniqueName: \"kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088896 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.088832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088896 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.088869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.088896 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:26.088884    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-serving-cert: secret "isvc-init-fail-2f3164-predictor-serving-cert" not found
Jul 27 21:36:26.089240 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:26.088948    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls podName:35fb89bc-4bba-49be-a0ea-26847e3c70b5 nodeName:}" failed. No retries permitted until 2026-07-27 21:36:26.588927623 +0000 UTC m=+1915.253552839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls") pod "isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5") : secret "isvc-init-fail-2f3164-predictor-serving-cert" not found
Jul 27 21:36:26.089348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.089247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.089486 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.089468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.089556 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.089534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.097696 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.097672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghxg\" (UniqueName: \"kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.593292 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.593249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.595672 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.595644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") pod \"isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") " pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.857592 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.857511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:26.992327 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:26.992302    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:26.994503 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:36:26.994475    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fb89bc_4bba_49be_a0ea_26847e3c70b5.slice/crio-1039d5a7f29d66835e1f94feee565bf11a5a77f56e0bce9f88f7140cbc78ddc6 WatchSource:0}: Error finding container 1039d5a7f29d66835e1f94feee565bf11a5a77f56e0bce9f88f7140cbc78ddc6: Status 404 returned error can't find the container with id 1039d5a7f29d66835e1f94feee565bf11a5a77f56e0bce9f88f7140cbc78ddc6
Jul 27 21:36:27.939861 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:27.939826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerStarted","Data":"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"}
Jul 27 21:36:27.940031 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:27.939886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerStarted","Data":"1039d5a7f29d66835e1f94feee565bf11a5a77f56e0bce9f88f7140cbc78ddc6"}
Jul 27 21:36:32.963530 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:32.963499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/0.log"
Jul 27 21:36:32.963888 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:32.963538    2580 generic.go:356] "Generic (PLEG): container finished" podID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerID="c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00" exitCode=1
Jul 27 21:36:32.963888 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:32.963617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerDied","Data":"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"}
Jul 27 21:36:33.969365 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:33.969336    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/0.log"
Jul 27 21:36:33.969780 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:33.969448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerStarted","Data":"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"}
Jul 27 21:36:35.945206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:35.945170    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:35.945668 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:35.945443    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer" containerID="cri-o://ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853" gracePeriod=30
Jul 27 21:36:36.910348 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.910312    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/1.log"
Jul 27 21:36:36.910812 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.910791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/0.log"
Jul 27 21:36:36.911982 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.911966    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:36.982496 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config\") pod \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") "
Jul 27 21:36:36.982496 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ghxg\" (UniqueName: \"kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg\") pod \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") "
Jul 27 21:36:36.982940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") pod \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") "
Jul 27 21:36:36.982940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location\") pod \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") "
Jul 27 21:36:36.982940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert\") pod \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\" (UID: \"35fb89bc-4bba-49be-a0ea-26847e3c70b5\") "
Jul 27 21:36:36.982940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35fb89bc-4bba-49be-a0ea-26847e3c70b5" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:36.982940 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35fb89bc-4bba-49be-a0ea-26847e3c70b5" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:36.983125 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.982968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35fb89bc-4bba-49be-a0ea-26847e3c70b5" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:36:36.985005 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.984961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35fb89bc-4bba-49be-a0ea-26847e3c70b5" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:36.985119 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg" (OuterVolumeSpecName: "kube-api-access-4ghxg") pod "35fb89bc-4bba-49be-a0ea-26847e3c70b5" (UID: "35fb89bc-4bba-49be-a0ea-26847e3c70b5"). InnerVolumeSpecName "kube-api-access-4ghxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:36.985119 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/1.log"
Jul 27 21:36:36.985525 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985511    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k_35fb89bc-4bba-49be-a0ea-26847e3c70b5/storage-initializer/0.log"
Jul 27 21:36:36.985618 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985543    2580 generic.go:356] "Generic (PLEG): container finished" podID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerID="ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853" exitCode=1
Jul 27 21:36:36.985618 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985599    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"
Jul 27 21:36:36.985723 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerDied","Data":"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"}
Jul 27 21:36:36.985782 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k" event={"ID":"35fb89bc-4bba-49be-a0ea-26847e3c70b5","Type":"ContainerDied","Data":"1039d5a7f29d66835e1f94feee565bf11a5a77f56e0bce9f88f7140cbc78ddc6"}
Jul 27 21:36:36.985836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:36.985781    2580 scope.go:117] "RemoveContainer" containerID="ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"
Jul 27 21:36:37.001100 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.001082    2580 scope.go:117] "RemoveContainer" containerID="c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"
Jul 27 21:36:37.008597 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.008578    2580 scope.go:117] "RemoveContainer" containerID="ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"
Jul 27 21:36:37.008846 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:37.008826    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853\": container with ID starting with ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853 not found: ID does not exist" containerID="ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"
Jul 27 21:36:37.008911 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.008854    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853"} err="failed to get container status \"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853\": rpc error: code = NotFound desc = could not find container \"ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853\": container with ID starting with ae0a78964bdef4aae9b92817fee18abf4036a667874efccd0cce14816353d853 not found: ID does not exist"
Jul 27 21:36:37.008911 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.008870    2580 scope.go:117] "RemoveContainer" containerID="c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"
Jul 27 21:36:37.009100 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:36:37.009086    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00\": container with ID starting with c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00 not found: ID does not exist" containerID="c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"
Jul 27 21:36:37.009145 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.009104    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00"} err="failed to get container status \"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00\": rpc error: code = NotFound desc = could not find container \"c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00\": container with ID starting with c3528faba8758a71313526f5b27b97b1dbb08c4fa77dd3ad09cc02b262d49e00 not found: ID does not exist"
Jul 27 21:36:37.027614 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.027584    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:37.031408 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.031384    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k"]
Jul 27 21:36:37.083831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.083803    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.083831 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.083825    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ghxg\" (UniqueName: \"kubernetes.io/projected/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kube-api-access-4ghxg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.084022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.083838    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35fb89bc-4bba-49be-a0ea-26847e3c70b5-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.084022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.083847    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35fb89bc-4bba-49be-a0ea-26847e3c70b5-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.084022 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.083856    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35fb89bc-4bba-49be-a0ea-26847e3c70b5-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:36:37.877489 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:37.877456    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" path="/var/lib/kubelet/pods/35fb89bc-4bba-49be-a0ea-26847e3c70b5/volumes"
Jul 27 21:36:41.159304 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159277    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:36:41.159669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.159669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.159669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159631    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.159669 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.159846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159694    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.159846 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.159789    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fb89bc-4bba-49be-a0ea-26847e3c70b5" containerName="storage-initializer"
Jul 27 21:36:41.162856 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.162836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.165203 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.165182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:36:41.165347 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.165301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:36:41.165425 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.165345    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:36:41.165425 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.165312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2483b-kube-rbac-proxy-sar-config\""
Jul 27 21:36:41.165526 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.165427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2483b-predictor-serving-cert\""
Jul 27 21:36:41.166240 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.166217    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:36:41.166374 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.166336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:36:41.180660 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.180639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:36:41.220334 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.220305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96zw\" (UniqueName: \"kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.220442 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.220355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.220442 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.220381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.220528 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.220479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.220528 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.220505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321315 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96zw\" (UniqueName: \"kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321432 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321544 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321590 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.321756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.322011 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.321988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.322126 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.322111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.324027 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.324002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.329868 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.329851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96zw\" (UniqueName: \"kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw\") pod \"raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") " pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.479962 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.479900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:41.610051 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:41.610028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:36:41.612563 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:36:41.612538    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b64c61_891c_4715_b061_75e4f2110a09.slice/crio-c90eab2c28e96a42fc49f333afa00f2526402767de30bd3e6cccabf1d70f47a3 WatchSource:0}: Error finding container c90eab2c28e96a42fc49f333afa00f2526402767de30bd3e6cccabf1d70f47a3: Status 404 returned error can't find the container with id c90eab2c28e96a42fc49f333afa00f2526402767de30bd3e6cccabf1d70f47a3
Jul 27 21:36:42.009633 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:42.009596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerStarted","Data":"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"}
Jul 27 21:36:42.009814 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:42.009652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerStarted","Data":"c90eab2c28e96a42fc49f333afa00f2526402767de30bd3e6cccabf1d70f47a3"}
Jul 27 21:36:46.028817 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:46.028785    2580 generic.go:356] "Generic (PLEG): container finished" podID="53b64c61-891c-4715-b061-75e4f2110a09" containerID="6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4" exitCode=0
Jul 27 21:36:46.029333 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:46.028838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerDied","Data":"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"}
Jul 27 21:36:47.035690 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:47.035655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerStarted","Data":"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"}
Jul 27 21:36:47.036062 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:47.035715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerStarted","Data":"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"}
Jul 27 21:36:47.036062 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:47.035887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:47.061465 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:47.061408    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podStartSLOduration=2.477012793 podStartE2EDuration="6.061390571s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584377778s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:47.058162981 +0000 UTC m=+1935.722788245" watchObservedRunningTime="2026-07-27 21:36:47.061390571 +0000 UTC m=+1935.726015810"
Jul 27 21:36:48.039854 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:48.039823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:48.041164 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:48.041139    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:36:49.044094 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:49.044049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:36:54.048844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:54.048816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:36:54.049308 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:36:54.049286    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:04.049336 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:04.049291    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:14.050133 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:14.050092    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:24.049293 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:24.049191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:34.049794 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:34.049753    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:44.049239 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:44.049196    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:54.049936 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:54.049900    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:37:55.872675 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:37:55.872634    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 21:38:05.877371 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:05.877342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:38:11.311910 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:11.311874    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:38:11.312632 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:11.312122    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container" containerID="cri-o://058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939" gracePeriod=30
Jul 27 21:38:11.312632 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:11.312322    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kube-rbac-proxy" containerID="cri-o://9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45" gracePeriod=30
Jul 27 21:38:12.416885 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:12.416856    2580 generic.go:356] "Generic (PLEG): container finished" podID="53b64c61-891c-4715-b061-75e4f2110a09" containerID="9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45" exitCode=2
Jul 27 21:38:12.417280 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:12.416909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerDied","Data":"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"}
Jul 27 21:38:14.044426 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:14.044384    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 27 21:38:15.766592 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.766569    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:38:15.792068 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert\") pod \"53b64c61-891c-4715-b061-75e4f2110a09\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") "
Jul 27 21:38:15.792199 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls\") pod \"53b64c61-891c-4715-b061-75e4f2110a09\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") "
Jul 27 21:38:15.792268 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location\") pod \"53b64c61-891c-4715-b061-75e4f2110a09\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") "
Jul 27 21:38:15.792318 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config\") pod \"53b64c61-891c-4715-b061-75e4f2110a09\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") "
Jul 27 21:38:15.792368 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n96zw\" (UniqueName: \"kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw\") pod \"53b64c61-891c-4715-b061-75e4f2110a09\" (UID: \"53b64c61-891c-4715-b061-75e4f2110a09\") "
Jul 27 21:38:15.792564 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53b64c61-891c-4715-b061-75e4f2110a09" (UID: "53b64c61-891c-4715-b061-75e4f2110a09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:38:15.792564 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792543    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53b64c61-891c-4715-b061-75e4f2110a09" (UID: "53b64c61-891c-4715-b061-75e4f2110a09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:15.792749 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.792620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53b64c61-891c-4715-b061-75e4f2110a09" (UID: "53b64c61-891c-4715-b061-75e4f2110a09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:38:15.794439 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.794410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw" (OuterVolumeSpecName: "kube-api-access-n96zw") pod "53b64c61-891c-4715-b061-75e4f2110a09" (UID: "53b64c61-891c-4715-b061-75e4f2110a09"). InnerVolumeSpecName "kube-api-access-n96zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:38:15.794439 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.794419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53b64c61-891c-4715-b061-75e4f2110a09" (UID: "53b64c61-891c-4715-b061-75e4f2110a09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:38:15.893631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.893604    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53b64c61-891c-4715-b061-75e4f2110a09-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:38:15.893631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.893628    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:38:15.893787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.893638    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96zw\" (UniqueName: \"kubernetes.io/projected/53b64c61-891c-4715-b061-75e4f2110a09-kube-api-access-n96zw\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:38:15.893787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.893646    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53b64c61-891c-4715-b061-75e4f2110a09-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:38:15.893787 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:15.893654    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53b64c61-891c-4715-b061-75e4f2110a09-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:38:16.436332 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.436292    2580 generic.go:356] "Generic (PLEG): container finished" podID="53b64c61-891c-4715-b061-75e4f2110a09" containerID="058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939" exitCode=0
Jul 27 21:38:16.436537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.436429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerDied","Data":"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"}
Jul 27 21:38:16.436537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.436438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"
Jul 27 21:38:16.436537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.436484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj" event={"ID":"53b64c61-891c-4715-b061-75e4f2110a09","Type":"ContainerDied","Data":"c90eab2c28e96a42fc49f333afa00f2526402767de30bd3e6cccabf1d70f47a3"}
Jul 27 21:38:16.436537 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.436512    2580 scope.go:117] "RemoveContainer" containerID="9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"
Jul 27 21:38:16.445195 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.445173    2580 scope.go:117] "RemoveContainer" containerID="058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"
Jul 27 21:38:16.451976 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.451952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:38:16.452477 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452458    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="storage-initializer"
Jul 27 21:38:16.452561 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="storage-initializer"
Jul 27 21:38:16.452561 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kube-rbac-proxy"
Jul 27 21:38:16.452561 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452500    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kube-rbac-proxy"
Jul 27 21:38:16.452561 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452521    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container"
Jul 27 21:38:16.452561 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452530    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container"
Jul 27 21:38:16.452756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452618    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kserve-container"
Jul 27 21:38:16.452756 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.452630    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b64c61-891c-4715-b061-75e4f2110a09" containerName="kube-rbac-proxy"
Jul 27 21:38:16.454200 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.454184    2580 scope.go:117] "RemoveContainer" containerID="6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"
Jul 27 21:38:16.457503 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.457482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.458238 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.458216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:38:16.459969 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.459943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-aff5e-predictor-serving-cert\""
Jul 27 21:38:16.460071 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-aff5e-kube-rbac-proxy-sar-config\""
Jul 27 21:38:16.460296 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:38:16.460296 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460288    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:38:16.460460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460297    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:38:16.460460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-cmm8z\""
Jul 27 21:38:16.460460 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.460297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:38:16.463134 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463113    2580 scope.go:117] "RemoveContainer" containerID="9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"
Jul 27 21:38:16.463493 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:38:16.463474    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45\": container with ID starting with 9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45 not found: ID does not exist" containerID="9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"
Jul 27 21:38:16.463579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463501    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45"} err="failed to get container status \"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45\": rpc error: code = NotFound desc = could not find container \"9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45\": container with ID starting with 9528f3e3040133e28252b13c4bba6000ef325eb2f7986791efe4c5a5527b8b45 not found: ID does not exist"
Jul 27 21:38:16.463579 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463519    2580 scope.go:117] "RemoveContainer" containerID="058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"
Jul 27 21:38:16.463789 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:38:16.463764    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939\": container with ID starting with 058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939 not found: ID does not exist" containerID="058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"
Jul 27 21:38:16.463879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939"} err="failed to get container status \"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939\": rpc error: code = NotFound desc = could not find container \"058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939\": container with ID starting with 058d6c218714d323eda5de391779d2bb2b03b8c52b0f8edae3a4a5874068a939 not found: ID does not exist"
Jul 27 21:38:16.463879 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463817    2580 scope.go:117] "RemoveContainer" containerID="6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"
Jul 27 21:38:16.464003 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.463954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:38:16.464114 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:38:16.464087    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4\": container with ID starting with 6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4 not found: ID does not exist" containerID="6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"
Jul 27 21:38:16.464154 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.464118    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4"} err="failed to get container status \"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4\": rpc error: code = NotFound desc = could not find container \"6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4\": container with ID starting with 6beaf6956eee08f3b205ed74f72915bbe63e5bd884ffbe425c71fd1fd8b641a4 not found: ID does not exist"
Jul 27 21:38:16.468266 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.468234    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj"]
Jul 27 21:38:16.497899 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.497877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.498010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.497930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.498010 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.498000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8lf\" (UniqueName: \"kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.498096 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.498022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.498096 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.498047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.598972 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.598945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.598992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8lf\" (UniqueName: \"kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599093 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599093 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:38:16.599088    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-serving-cert: secret "raw-sklearn-runtime-aff5e-predictor-serving-cert" not found
Jul 27 21:38:16.599394 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:38:16.599147    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls podName:7278894f-e5cd-4d16-88cc-5a83489b7821 nodeName:}" failed. No retries permitted until 2026-07-27 21:38:17.099132142 +0000 UTC m=+2025.763757362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls") pod "raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821") : secret "raw-sklearn-runtime-aff5e-predictor-serving-cert" not found
Jul 27 21:38:16.599463 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599692 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.599735 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.599707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:16.607533 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:16.607514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8lf\" (UniqueName: \"kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:17.104143 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:17.104113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:17.106603 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:17.106573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") pod \"raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:17.370855 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:17.370759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:17.506501 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:17.506474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:38:17.509031 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:38:17.508997    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7278894f_e5cd_4d16_88cc_5a83489b7821.slice/crio-e6790229ca592300ad4f46312022ec672d088a6ae0bcd71dacdd3e27f736a34d WatchSource:0}: Error finding container e6790229ca592300ad4f46312022ec672d088a6ae0bcd71dacdd3e27f736a34d: Status 404 returned error can't find the container with id e6790229ca592300ad4f46312022ec672d088a6ae0bcd71dacdd3e27f736a34d
Jul 27 21:38:17.878601 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:17.878566    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b64c61-891c-4715-b061-75e4f2110a09" path="/var/lib/kubelet/pods/53b64c61-891c-4715-b061-75e4f2110a09/volumes"
Jul 27 21:38:18.448589 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:18.448551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerStarted","Data":"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"}
Jul 27 21:38:18.448987 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:18.448606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerStarted","Data":"e6790229ca592300ad4f46312022ec672d088a6ae0bcd71dacdd3e27f736a34d"}
Jul 27 21:38:21.465409 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:21.465376    2580 generic.go:356] "Generic (PLEG): container finished" podID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerID="f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d" exitCode=0
Jul 27 21:38:21.465773 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:21.465449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerDied","Data":"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"}
Jul 27 21:38:22.472114 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:22.472082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerStarted","Data":"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"}
Jul 27 21:38:22.472523 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:22.472139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerStarted","Data":"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"}
Jul 27 21:38:22.472523 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:22.472391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:22.495198 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:22.495157    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podStartSLOduration=2.745498902 podStartE2EDuration="6.49514332s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.749644418s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:22.49445808 +0000 UTC m=+2031.159083317" watchObservedRunningTime="2026-07-27 21:38:22.49514332 +0000 UTC m=+2031.159768557"
Jul 27 21:38:23.476834 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:23.476804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:23.478025 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:23.478001    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:38:24.481609 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:24.481567    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:38:29.486251 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:29.486216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:38:29.486840 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:29.486804    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:38:39.486907 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:39.486864    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:38:49.487729 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:49.487689    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:38:59.487182 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:38:59.487101    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:39:09.487517 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:09.487481    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:39:19.486930 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:19.486893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:39:29.486844 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:29.486810    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:39:39.487858 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:39.487828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:39:46.577492 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:46.577461    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:39:46.578222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:46.577692    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" containerID="cri-o://fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1" gracePeriod=30
Jul 27 21:39:46.578222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:46.577773    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kube-rbac-proxy" containerID="cri-o://6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634" gracePeriod=30
Jul 27 21:39:46.858322 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:46.858224    2580 generic.go:356] "Generic (PLEG): container finished" podID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerID="6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634" exitCode=2
Jul 27 21:39:46.858322 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:46.858305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerDied","Data":"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"}
Jul 27 21:39:49.482504 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:49.482468    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 27 21:39:49.489172 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:49.489131    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 27 21:39:50.724336 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.724314    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:39:50.882581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.882499    2580 generic.go:356] "Generic (PLEG): container finished" podID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerID="fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1" exitCode=0
Jul 27 21:39:50.882581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.882557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"
Jul 27 21:39:50.882581 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.882563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerDied","Data":"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"}
Jul 27 21:39:50.882795 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.882614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b" event={"ID":"7278894f-e5cd-4d16-88cc-5a83489b7821","Type":"ContainerDied","Data":"e6790229ca592300ad4f46312022ec672d088a6ae0bcd71dacdd3e27f736a34d"}
Jul 27 21:39:50.882795 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.882645    2580 scope.go:117] "RemoveContainer" containerID="6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"
Jul 27 21:39:50.886549 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert\") pod \"7278894f-e5cd-4d16-88cc-5a83489b7821\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") "
Jul 27 21:39:50.886684 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config\") pod \"7278894f-e5cd-4d16-88cc-5a83489b7821\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") "
Jul 27 21:39:50.886684 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb8lf\" (UniqueName: \"kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf\") pod \"7278894f-e5cd-4d16-88cc-5a83489b7821\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") "
Jul 27 21:39:50.886684 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location\") pod \"7278894f-e5cd-4d16-88cc-5a83489b7821\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") "
Jul 27 21:39:50.886857 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") pod \"7278894f-e5cd-4d16-88cc-5a83489b7821\" (UID: \"7278894f-e5cd-4d16-88cc-5a83489b7821\") "
Jul 27 21:39:50.886911 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886868    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7278894f-e5cd-4d16-88cc-5a83489b7821" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:39:50.886965 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886948    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-cabundle-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:39:50.887026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7278894f-e5cd-4d16-88cc-5a83489b7821" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:50.887026 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.886999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7278894f-e5cd-4d16-88cc-5a83489b7821" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:39:50.888968 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.888936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf" (OuterVolumeSpecName: "kube-api-access-vb8lf") pod "7278894f-e5cd-4d16-88cc-5a83489b7821" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821"). InnerVolumeSpecName "kube-api-access-vb8lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:39:50.889069 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.889034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7278894f-e5cd-4d16-88cc-5a83489b7821" (UID: "7278894f-e5cd-4d16-88cc-5a83489b7821"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:39:50.892330 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.892311    2580 scope.go:117] "RemoveContainer" containerID="fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"
Jul 27 21:39:50.904317 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.904284    2580 scope.go:117] "RemoveContainer" containerID="f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"
Jul 27 21:39:50.911511 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.911497    2580 scope.go:117] "RemoveContainer" containerID="6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"
Jul 27 21:39:50.911778 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:39:50.911760    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634\": container with ID starting with 6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634 not found: ID does not exist" containerID="6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"
Jul 27 21:39:50.911836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.911788    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634"} err="failed to get container status \"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634\": rpc error: code = NotFound desc = could not find container \"6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634\": container with ID starting with 6977bfb4dcc38e6b9f30ff123f02cb184137312e87b37307d6a5367712e9a634 not found: ID does not exist"
Jul 27 21:39:50.911836 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.911804    2580 scope.go:117] "RemoveContainer" containerID="fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"
Jul 27 21:39:50.912081 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:39:50.912064    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1\": container with ID starting with fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1 not found: ID does not exist" containerID="fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"
Jul 27 21:39:50.912140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.912087    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1"} err="failed to get container status \"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1\": rpc error: code = NotFound desc = could not find container \"fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1\": container with ID starting with fd837707676c4ed3d65e8aa10d04db6056801103eeb0bfa2c8e33c319f3264f1 not found: ID does not exist"
Jul 27 21:39:50.912140 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.912102    2580 scope.go:117] "RemoveContainer" containerID="f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"
Jul 27 21:39:50.912330 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:39:50.912315    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d\": container with ID starting with f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d not found: ID does not exist" containerID="f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"
Jul 27 21:39:50.912379 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.912334    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d"} err="failed to get container status \"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d\": rpc error: code = NotFound desc = could not find container \"f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d\": container with ID starting with f53e65815e9efe96859f848db04588620863a2ce2f8b7d2ba8a551ad0ce75e3d not found: ID does not exist"
Jul 27 21:39:50.988359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.988321    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:39:50.988359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.988354    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8lf\" (UniqueName: \"kubernetes.io/projected/7278894f-e5cd-4d16-88cc-5a83489b7821-kube-api-access-vb8lf\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:39:50.988359 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.988364    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7278894f-e5cd-4d16-88cc-5a83489b7821-kserve-provision-location\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:39:50.988546 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:50.988373    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7278894f-e5cd-4d16-88cc-5a83489b7821-proxy-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:39:51.203807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:51.203777    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:39:51.206705 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:51.206683    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b"]
Jul 27 21:39:51.878589 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:51.878559    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" path="/var/lib/kubelet/pods/7278894f-e5cd-4d16-88cc-5a83489b7821/volumes"
Jul 27 21:39:52.397249 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vdt2n/must-gather-7dxfm"]
Jul 27 21:39:52.397575 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397563    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="storage-initializer"
Jul 27 21:39:52.397623 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="storage-initializer"
Jul 27 21:39:52.397623 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container"
Jul 27 21:39:52.397623 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397593    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container"
Jul 27 21:39:52.397623 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397601    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kube-rbac-proxy"
Jul 27 21:39:52.397623 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397607    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kube-rbac-proxy"
Jul 27 21:39:52.397775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kube-rbac-proxy"
Jul 27 21:39:52.397775 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.397667    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7278894f-e5cd-4d16-88cc-5a83489b7821" containerName="kserve-container"
Jul 27 21:39:52.400764 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.400746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.403103 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.403080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vdt2n\"/\"default-dockercfg-6gtff\""
Jul 27 21:39:52.403222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.403166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdt2n\"/\"kube-root-ca.crt\""
Jul 27 21:39:52.403222 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.403167    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vdt2n\"/\"openshift-service-ca.crt\""
Jul 27 21:39:52.411377 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.411357    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdt2n/must-gather-7dxfm"]
Jul 27 21:39:52.500655 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.500627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.500754 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.500667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vlz\" (UniqueName: \"kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.601404 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.601378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.601565 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.601432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vlz\" (UniqueName: \"kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.601688 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.601671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.610507 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.610484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vlz\" (UniqueName: \"kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz\") pod \"must-gather-7dxfm\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") " pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.716538 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.716476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:39:52.844110 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.844086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vdt2n/must-gather-7dxfm"]
Jul 27 21:39:52.846272 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:39:52.846235    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc8b284_0da0_43da_989a_6463dd0899ea.slice/crio-0310d8a73fa2f0fb73d2ec3ec5a3a2a6fc55a5342fd1bdf0f62ace2b0f3553eb WatchSource:0}: Error finding container 0310d8a73fa2f0fb73d2ec3ec5a3a2a6fc55a5342fd1bdf0f62ace2b0f3553eb: Status 404 returned error can't find the container with id 0310d8a73fa2f0fb73d2ec3ec5a3a2a6fc55a5342fd1bdf0f62ace2b0f3553eb
Jul 27 21:39:52.848091 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.848077    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:39:52.892204 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:52.892175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" event={"ID":"8bc8b284-0da0-43da-989a-6463dd0899ea","Type":"ContainerStarted","Data":"0310d8a73fa2f0fb73d2ec3ec5a3a2a6fc55a5342fd1bdf0f62ace2b0f3553eb"}
Jul 27 21:39:59.932372 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:59.932328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" event={"ID":"8bc8b284-0da0-43da-989a-6463dd0899ea","Type":"ContainerStarted","Data":"3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1"}
Jul 27 21:39:59.932372 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:59.932377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" event={"ID":"8bc8b284-0da0-43da-989a-6463dd0899ea","Type":"ContainerStarted","Data":"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"}
Jul 27 21:39:59.946655 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:39:59.946609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" podStartSLOduration=1.787078185 podStartE2EDuration="7.946594735s" totalImagesPullingTime="6.15951655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:59.94543131 +0000 UTC m=+2128.610056548" watchObservedRunningTime="2026-07-27 21:39:59.946594735 +0000 UTC m=+2128.611219974"
Jul 27 21:40:21.041627 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:21.041596    2580 generic.go:356] "Generic (PLEG): container finished" podID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerID="09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a" exitCode=0
Jul 27 21:40:21.042017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:21.041643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" event={"ID":"8bc8b284-0da0-43da-989a-6463dd0899ea","Type":"ContainerDied","Data":"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"}
Jul 27 21:40:21.042017 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:21.041932    2580 scope.go:117] "RemoveContainer" containerID="09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"
Jul 27 21:40:21.961387 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:21.961353    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vdt2n_must-gather-7dxfm_8bc8b284-0da0-43da-989a-6463dd0899ea/gather/0.log"
Jul 27 21:40:25.196620 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:25.196587    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zhdtx_2eec02d3-3654-484f-9f70-60fbac97a123/global-pull-secret-syncer/0.log"
Jul 27 21:40:25.284120 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:25.284094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-59lqd_2dc57acf-9d20-4b82-b522-8a44e559a4d4/konnectivity-agent/0.log"
Jul 27 21:40:25.343219 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:25.343194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-253.ec2.internal_f1c1da79ad55c5026a8b619f1f45eec5/haproxy/0.log"
Jul 27 21:40:27.360472 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.360438    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vdt2n/must-gather-7dxfm"]
Jul 27 21:40:27.360964 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.360672    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vdt2n/must-gather-7dxfm" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="copy" containerID="cri-o://3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1" gracePeriod=2
Jul 27 21:40:27.363189 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.363162    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vdt2n/must-gather-7dxfm"]
Jul 27 21:40:27.363364 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:40:27.363335    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7dxfm\" is forbidden: User \"system:node:ip-10-0-131-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vdt2n\": no relationship found between node 'ip-10-0-131-253.ec2.internal' and this object" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:40:27.593796 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.593775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vdt2n_must-gather-7dxfm_8bc8b284-0da0-43da-989a-6463dd0899ea/copy/0.log"
Jul 27 21:40:27.595180 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.595164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:40:27.597093 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:40:27.597070    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7dxfm\" is forbidden: User \"system:node:ip-10-0-131-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vdt2n\": no relationship found between node 'ip-10-0-131-253.ec2.internal' and this object" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:40:27.696344 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.696252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output\") pod \"8bc8b284-0da0-43da-989a-6463dd0899ea\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") "
Jul 27 21:40:27.696344 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.696306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4vlz\" (UniqueName: \"kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz\") pod \"8bc8b284-0da0-43da-989a-6463dd0899ea\" (UID: \"8bc8b284-0da0-43da-989a-6463dd0899ea\") "
Jul 27 21:40:27.697661 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.697620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8bc8b284-0da0-43da-989a-6463dd0899ea" (UID: "8bc8b284-0da0-43da-989a-6463dd0899ea"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:27.698685 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.698658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz" (OuterVolumeSpecName: "kube-api-access-q4vlz") pod "8bc8b284-0da0-43da-989a-6463dd0899ea" (UID: "8bc8b284-0da0-43da-989a-6463dd0899ea"). InnerVolumeSpecName "kube-api-access-q4vlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:40:27.797685 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.797651    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bc8b284-0da0-43da-989a-6463dd0899ea-must-gather-output\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:40:27.797685 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.797685    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vlz\" (UniqueName: \"kubernetes.io/projected/8bc8b284-0da0-43da-989a-6463dd0899ea-kube-api-access-q4vlz\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\""
Jul 27 21:40:27.877799 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:27.877772    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" path="/var/lib/kubelet/pods/8bc8b284-0da0-43da-989a-6463dd0899ea/volumes"
Jul 27 21:40:28.074856 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.074825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vdt2n_must-gather-7dxfm_8bc8b284-0da0-43da-989a-6463dd0899ea/copy/0.log"
Jul 27 21:40:28.075155 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.075134    2580 generic.go:356] "Generic (PLEG): container finished" podID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerID="3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1" exitCode=143
Jul 27 21:40:28.075204 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.075175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vdt2n/must-gather-7dxfm"
Jul 27 21:40:28.075248 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.075230    2580 scope.go:117] "RemoveContainer" containerID="3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1"
Jul 27 21:40:28.083347 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.083321    2580 scope.go:117] "RemoveContainer" containerID="09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"
Jul 27 21:40:28.098705 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.098688    2580 scope.go:117] "RemoveContainer" containerID="3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1"
Jul 27 21:40:28.098931 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:40:28.098914    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1\": container with ID starting with 3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1 not found: ID does not exist" containerID="3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1"
Jul 27 21:40:28.098993 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.098938    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1"} err="failed to get container status \"3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1\": rpc error: code = NotFound desc = could not find container \"3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1\": container with ID starting with 3f98cfe2c2a44732ec2369d35e29f2e52a9c42503411690826f2f5b6f57d98f1 not found: ID does not exist"
Jul 27 21:40:28.098993 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.098957    2580 scope.go:117] "RemoveContainer" containerID="09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"
Jul 27 21:40:28.099201 ip-10-0-131-253 kubenswrapper[2580]: E0727 21:40:28.099183    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a\": container with ID starting with 09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a not found: ID does not exist" containerID="09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"
Jul 27 21:40:28.099276 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:28.099212    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a"} err="failed to get container status \"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a\": rpc error: code = NotFound desc = could not find container \"09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a\": container with ID starting with 09217745f22f97401611a38615fdd1307b433adfd5ff82769be0f76b34b20e6a not found: ID does not exist"
Jul 27 21:40:29.103698 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.103667    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-np7mb_41404943-3602-4dc6-8231-c0574cda3c58/kube-state-metrics/0.log"
Jul 27 21:40:29.132428 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.132396    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-np7mb_41404943-3602-4dc6-8231-c0574cda3c58/kube-rbac-proxy-main/0.log"
Jul 27 21:40:29.166206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.166185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-np7mb_41404943-3602-4dc6-8231-c0574cda3c58/kube-rbac-proxy-self/0.log"
Jul 27 21:40:29.277983 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.277956    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-286sf_ffd32ae6-a080-40f3-a119-137a013c6d4f/node-exporter/0.log"
Jul 27 21:40:29.303900 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.303872    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-286sf_ffd32ae6-a080-40f3-a119-137a013c6d4f/kube-rbac-proxy/0.log"
Jul 27 21:40:29.331631 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.331609    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-286sf_ffd32ae6-a080-40f3-a119-137a013c6d4f/init-textfile/0.log"
Jul 27 21:40:29.553864 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.553843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vd965_d671ec71-956f-47fb-af06-b3f6c5cc302e/kube-rbac-proxy-main/0.log"
Jul 27 21:40:29.579867 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.579847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vd965_d671ec71-956f-47fb-af06-b3f6c5cc302e/kube-rbac-proxy-self/0.log"
Jul 27 21:40:29.607171 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.607142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vd965_d671ec71-956f-47fb-af06-b3f6c5cc302e/openshift-state-metrics/0.log"
Jul 27 21:40:29.836555 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.836479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-pnpqk_055078c0-d43c-49d8-b799-090f0246e0b0/prometheus-operator/0.log"
Jul 27 21:40:29.856205 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.856183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-pnpqk_055078c0-d43c-49d8-b799-090f0246e0b0/kube-rbac-proxy/0.log"
Jul 27 21:40:29.895470 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:29.895443    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-q6v6g_0552e627-a250-483e-a98a-91e17d0c0776/prometheus-operator-admission-webhook/0.log"
Jul 27 21:40:30.065206 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.065178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/thanos-query/0.log"
Jul 27 21:40:30.092223 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.092154    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/kube-rbac-proxy-web/0.log"
Jul 27 21:40:30.117542 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.117523    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/kube-rbac-proxy/0.log"
Jul 27 21:40:30.152241 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.152218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/prom-label-proxy/0.log"
Jul 27 21:40:30.177499 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.177482    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/kube-rbac-proxy-rules/0.log"
Jul 27 21:40:30.203316 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:30.203299    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5d496fb6-v855k_c43bffdf-7834-4311-8bcd-7c1fe7ac8943/kube-rbac-proxy-metrics/0.log"
Jul 27 21:40:32.589808 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:32.589779    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d766c59c9-vsgn7_e42b3955-0bb4-4c6d-90c5-a5f9ed3ea497/console/0.log"
Jul 27 21:40:32.625895 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:32.625866    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-zv4gm_7a5f8ff0-9f6a-463b-b704-22086fd8b8da/download-server/0.log"
Jul 27 21:40:33.129908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.129878    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-csc2v_1ae13130-fc45-41e6-8dc9-65418018048b/volume-data-source-validator/0.log"
Jul 27 21:40:33.440853 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.440774    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"]
Jul 27 21:40:33.441114 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441101    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="gather"
Jul 27 21:40:33.441169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="gather"
Jul 27 21:40:33.441169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441122    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="copy"
Jul 27 21:40:33.441169 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="copy"
Jul 27 21:40:33.441277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="copy"
Jul 27 21:40:33.441277 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.441198    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc8b284-0da0-43da-989a-6463dd0899ea" containerName="gather"
Jul 27 21:40:33.448104 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.448084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.450942 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.450672    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbjts\"/\"kube-root-ca.crt\""
Jul 27 21:40:33.451807 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.451785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cbjts\"/\"default-dockercfg-5jhdr\""
Jul 27 21:40:33.451908 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.451830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbjts\"/\"openshift-service-ca.crt\""
Jul 27 21:40:33.452758 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.452739    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"]
Jul 27 21:40:33.548468 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.548436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-sys\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.548650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.548488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvrn\" (UniqueName: \"kubernetes.io/projected/438abe7c-a320-41b4-b46b-ccdc4352b9db-kube-api-access-xbvrn\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.548650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.548552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-lib-modules\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.548650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.548572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-proc\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.548650 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.548588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-podres\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649106 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-lib-modules\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-proc\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-podres\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-sys\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvrn\" (UniqueName: \"kubernetes.io/projected/438abe7c-a320-41b4-b46b-ccdc4352b9db-kube-api-access-xbvrn\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-sys\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-proc\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-lib-modules\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.649516 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.649342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/438abe7c-a320-41b4-b46b-ccdc4352b9db-podres\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.658244 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.658221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvrn\" (UniqueName: \"kubernetes.io/projected/438abe7c-a320-41b4-b46b-ccdc4352b9db-kube-api-access-xbvrn\") pod \"perf-node-gather-daemonset-f6x98\" (UID: \"438abe7c-a320-41b4-b46b-ccdc4352b9db\") " pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.758979 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.758900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:33.887281 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:33.887095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"]
Jul 27 21:40:33.889393 ip-10-0-131-253 kubenswrapper[2580]: W0727 21:40:33.889368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod438abe7c_a320_41b4_b46b_ccdc4352b9db.slice/crio-9fc5349bcef899b04202e3326ac1e7eb5f5591ad5e732f8ddafa912ee7a959d7 WatchSource:0}: Error finding container 9fc5349bcef899b04202e3326ac1e7eb5f5591ad5e732f8ddafa912ee7a959d7: Status 404 returned error can't find the container with id 9fc5349bcef899b04202e3326ac1e7eb5f5591ad5e732f8ddafa912ee7a959d7
Jul 27 21:40:34.080886 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.080830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmpfv_9ae68659-399b-4891-a9f8-bf55fae841a8/dns/0.log"
Jul 27 21:40:34.101707 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.101682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmpfv_9ae68659-399b-4891-a9f8-bf55fae841a8/kube-rbac-proxy/0.log"
Jul 27 21:40:34.103863 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.103833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98" event={"ID":"438abe7c-a320-41b4-b46b-ccdc4352b9db","Type":"ContainerStarted","Data":"7575c0064bc5a0d9a99ec4e33b71f4b40a96a09459183be9874f604c175148bd"}
Jul 27 21:40:34.104008 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.103886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98" event={"ID":"438abe7c-a320-41b4-b46b-ccdc4352b9db","Type":"ContainerStarted","Data":"9fc5349bcef899b04202e3326ac1e7eb5f5591ad5e732f8ddafa912ee7a959d7"}
Jul 27 21:40:34.104008 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.103915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:34.120377 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.120336    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98" podStartSLOduration=1.1203219660000001 podStartE2EDuration="1.120321966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:34.118901673 +0000 UTC m=+2162.783526933" watchObservedRunningTime="2026-07-27 21:40:34.120321966 +0000 UTC m=+2162.784947204"
Jul 27 21:40:34.271099 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.271065    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-856bp_1796f663-d2a4-4f37-9442-3ea4ead2efc4/dns-node-resolver/0.log"
Jul 27 21:40:34.763550 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.763500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fb7cf5bd4-w5xsz_c034b8cc-a392-4070-9c43-679e7ac36d59/registry/0.log"
Jul 27 21:40:34.785474 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:34.785445    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hf45z_8e468562-5b4a-418b-8ae4-360be809fde2/node-ca/0.log"
Jul 27 21:40:35.552790 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:35.552757    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cb9c7cff9-gn874_ff98cefc-9b31-4473-bd21-68d1dc2b191d/router/0.log"
Jul 27 21:40:35.937083 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:35.937057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6skq5_a6506781-41f2-4f90-ad2b-a0d7f9ccc627/serve-healthcheck-canary/0.log"
Jul 27 21:40:36.303788 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:36.303736    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-wtrtk_f8ca8d50-1f85-489a-a938-779325f199b1/insights-operator/1.log"
Jul 27 21:40:36.304150 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:36.304132    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-wtrtk_f8ca8d50-1f85-489a-a938-779325f199b1/insights-operator/0.log"
Jul 27 21:40:36.325922 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:36.325897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxbp_445ace8f-0db8-449f-916a-0daaa96c9691/kube-rbac-proxy/0.log"
Jul 27 21:40:36.347328 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:36.347305    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxbp_445ace8f-0db8-449f-916a-0daaa96c9691/exporter/0.log"
Jul 27 21:40:36.369649 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:36.369612    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxbp_445ace8f-0db8-449f-916a-0daaa96c9691/extractor/0.log"
Jul 27 21:40:38.719321 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:38.719287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-vmkgt_c1c06c1c-e051-48f3-b141-4c88ef5eeb61/server/0.log"
Jul 27 21:40:38.918725 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:38.918690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-pwlgz_157aae4d-5ff7-4198-a0fb-c9f1279e6fef/manager/0.log"
Jul 27 21:40:38.995141 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:38.995058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-kxfcc_36e7812e-1625-4f99-895f-acb1f45b6e1a/seaweedfs/0.log"
Jul 27 21:40:40.118429 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:40.118403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cbjts/perf-node-gather-daemonset-f6x98"
Jul 27 21:40:44.145683 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.145646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4dppf_1d1049b4-69ae-445c-bae0-4ca9de4a6cbb/kube-multus/0.log"
Jul 27 21:40:44.540994 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.540970    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:40:44.565390 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.565370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/egress-router-binary-copy/0.log"
Jul 27 21:40:44.590376 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.590352    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/cni-plugins/0.log"
Jul 27 21:40:44.613626 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.613598    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/bond-cni-plugin/0.log"
Jul 27 21:40:44.638959 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.638929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/routeoverride-cni/0.log"
Jul 27 21:40:44.665934 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.665915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/whereabouts-cni-bincopy/0.log"
Jul 27 21:40:44.687633 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.687571    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzkd7_63836abf-191c-49be-ab1c-98f96564118d/whereabouts-cni/0.log"
Jul 27 21:40:44.862521 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.862491    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-phzrj_9a84e1cd-17c3-43dc-b4d7-47c93e63686f/network-metrics-daemon/0.log"
Jul 27 21:40:44.880515 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:44.880492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-phzrj_9a84e1cd-17c3-43dc-b4d7-47c93e63686f/kube-rbac-proxy/0.log"
Jul 27 21:40:46.002484 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.002454    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/ovn-controller/0.log"
Jul 27 21:40:46.024163 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.024138    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/ovn-acl-logging/0.log"
Jul 27 21:40:46.050115 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.050090    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/kube-rbac-proxy-node/0.log"
Jul 27 21:40:46.072907 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.072883    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 21:40:46.090856 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.090832    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/northd/0.log"
Jul 27 21:40:46.111621 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.111601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/nbdb/0.log"
Jul 27 21:40:46.133509 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.133487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/sbdb/0.log"
Jul 27 21:40:46.197976 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:46.197953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk54c_147e8328-dcf4-4551-bdb1-d8b84bd08858/ovnkube-controller/0.log"
Jul 27 21:40:47.251382 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:47.251346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-sbdd7_18926344-1fbe-4113-b076-97a3ee1f0705/check-endpoints/0.log"
Jul 27 21:40:47.341708 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:47.341679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rcjn8_1ea324c2-61c8-4c0d-af90-6f1aa62a2fcb/network-check-target-container/0.log"
Jul 27 21:40:48.281706 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:48.281679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zk6x4_b9f92be2-dda7-4dd3-a43c-0f4ca41ecb2c/iptables-alerter/0.log"
Jul 27 21:40:48.911744 ip-10-0-131-253 kubenswrapper[2580]: I0727 21:40:48.911716    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9wzmp_dac5a78f-0a74-45bb-88f4-5a83d35b14ff/tuned/0.log"