Sep 09 20:11:00.512400 ip-10-0-143-149 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:00.902898 ip-10-0-143-149 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:00.902898 ip-10-0-143-149 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:00.902898 ip-10-0-143-149 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:00.902898 ip-10-0-143-149 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:00.902898 ip-10-0-143-149 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:00.904725 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.904630    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:00.909200 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909179    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.909200 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909199    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.909200 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909202    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909205    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909209    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909212    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909215    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909218    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909221    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909224    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909227    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909229    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909233    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909237    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909240    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909242    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909245    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909248    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909250    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909253    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909256    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909259    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.909343 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909261    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909264    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909266    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909269    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909272    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909275    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909278    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909280    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909283    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909285    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909288    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909290    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909293    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909295    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909299    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909302    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909305    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909308    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909312    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.909827 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909316    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909319    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909322    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909325    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909327    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909330    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909333    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909335    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909338    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909340    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909343    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909346    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909349    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909352    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909354    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909357    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909360    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909363    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909365    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909368    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.910340 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909371    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909374    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909376    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909378    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909381    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909384    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909386    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909389    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909392    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909396    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909399    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909401    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909404    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909406    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909409    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909411    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909415    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909418    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909420    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.910831 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909423    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909428    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909431    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909434    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909438    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909441    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909443    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909446    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909449    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909451    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909454    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909456    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909459    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909463    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909466    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909470    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909472    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909475    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.909478    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.911310 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910970    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910978    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910983    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910987    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910990    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910994    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.910997    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911008    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911012    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911014    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911017    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911020    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911022    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911025    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911028    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911032    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911035    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911038    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911040    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.911769 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911043    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911046    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911048    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911051    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911054    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911069    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911072    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911075    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911078    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911081    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911083    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911086    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911089    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911091    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911094    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911097    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911099    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911102    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911105    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911108    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.912298 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911111    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911113    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911116    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911119    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911121    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911124    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911127    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911130    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911134    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911137    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911141    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911144    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911147    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911150    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911153    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911156    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911159    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911161    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911164    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911166    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.912788 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911169    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911171    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911174    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911177    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911179    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911182    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911184    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911187    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911189    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911193    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911196    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911198    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911201    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911204    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911207    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911209    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911212    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911215    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911217    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911220    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.913305 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911224    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911227    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911229    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911232    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911235    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911237    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911240    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911243    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911245    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911248    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911251    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911253    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911256    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911258    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911261    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911263    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911266    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911269    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911271    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911274    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.913823 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911701    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911706    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911709    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911712    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911716    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911719    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911722    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911725    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911729    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911734    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911736    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911739    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911742    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911744    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911746    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911749    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911752    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911754    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911757    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911759    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.914440 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911762    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911764    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911767    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911770    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911772    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911775    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911778    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911781    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911783    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911786    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911789    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911791    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911794    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911797    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911799    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911801    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911804    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911808    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911811    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911813    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.914945 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911816    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911819    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911821    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911824    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911826    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911829    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911831    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911834    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911836    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911838    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911841    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911843    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911846    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911848    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911851    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911853    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911855    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911858    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911860    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.915456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911863    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911868    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911871    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911874    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911876    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911879    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911882    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911884    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911886    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911889    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911892    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911896    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911898    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911900    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911903    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911905    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911908    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911910    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911913    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.915915 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911916    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911918    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911921    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911923    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911926    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911928    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911931    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911933    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911936    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911938    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911942    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911944    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911947    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911950    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911952    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911955    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911958    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911960    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911963    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911965    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.916388 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.911969    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912068    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912122    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912131    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912135    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912141    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912145    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912149    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912154    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912157    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912160    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912164    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912167    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912170    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912174    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912176    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912182    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912185    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912188    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912193    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912196    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912199    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912202    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912206    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:00.916874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912210    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912213    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912216    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912220    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912223    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912226    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912229    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912232    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912235    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912239    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912243    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912245    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912248    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912251    2564 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912255    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912260    2564 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912263    2564 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912266    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912269    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912272    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912276    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912279    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912282    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912285    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912289    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:00.917505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912292    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912295    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912298    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912301    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912305    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912308    2564 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912312    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912315    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912318    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912321    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912325    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912328    2564 flags.go:64] FLAG: --help="false"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912330    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-143-149.ec2.internal"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912334    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912337    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912341    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912345    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912348    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912351    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912354    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912357    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912360    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912363    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912367    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:00.918151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912369    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912372    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912375    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912378    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912381    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912384    2564 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912387    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912392    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912395    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912401    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912404    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912407    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912410    2564 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912413    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912416    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912419    2564 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912423    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912427    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912431    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912435    2564 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912438    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912442    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912445    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912449    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912452    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:00.918745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912456    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912459    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912467    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912471    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912474    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912477    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912481    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912487    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912490    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912494    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912497    2564 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912500    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912504    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e80c578c0a9b65d"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912507    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912511    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912514    2564 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912517    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912521    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912524    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912526    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912529    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912534    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912537    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912540    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912543    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:00.919388 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912546    2564 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912549    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912552    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912555    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912558    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912562    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912566    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912570    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912573    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912576    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912579    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912581    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912584    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912588    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912594    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912597    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912602    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912605    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912608    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912612    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912615    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912619    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912622    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912625    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912628    2564 flags.go:64] FLAG: --v="2"
Sep 09 20:11:00.920022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912631    2564 flags.go:64] FLAG: --version="false"
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912636    2564 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912639    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.912643    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912773    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912777    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912780    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912783    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912786    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912789    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912792    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912795    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912798    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912802    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912805    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912808    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912811    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912813    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912816    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912818    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.920665 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912821    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912824    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912828    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912830    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912833    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912836    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912838    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912841    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912845    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912847    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912850    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912852    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912855    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912857    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912860    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912862    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912865    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912867    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912870    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912872    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912876    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.921213 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912878    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912881    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912883    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912886    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912891    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912894    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912896    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912899    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912902    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912907    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912910    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912912    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912916    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912919    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912922    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912925    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912927    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912930    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912932    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.921742 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912936    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912939    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912942    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912944    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912947    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912949    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912951    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912954    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912957    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912959    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912963    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912966    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912969    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912972    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912975    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912977    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912979    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912983    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912986    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.922239 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912988    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912991    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912993    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.912996    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913000    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913003    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913006    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913009    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913011    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913014    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913017    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913020    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913022    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913026    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913029    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913032    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913034    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913037    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913039    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913042    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.922719 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913044    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913047    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913049    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.913052    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.913662    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.920944    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.920964    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.921625    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.921635    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921703    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921710    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921713    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921716    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921719    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.923245 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921722    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921725    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921727    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921730    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921733    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921735    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921738    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921741    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921746    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921750    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921755    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921758    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921761    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921764    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921766    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921769    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921772    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921776    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921780    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921783    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.923597 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921786    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921788    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921791    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921794    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921799    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921802    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921804    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921807    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921810    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921812    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921815    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921817    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921821    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921825    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921829    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921833    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921835    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921838    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921840    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921843    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.924103 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921846    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921849    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921852    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921854    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921857    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921859    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921862    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921864    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921867    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921870    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921872    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921875    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921878    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921880    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921883    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921886    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921889    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921893    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921896    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921900    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.924678 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921904    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921908    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921911    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921914    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921917    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921919    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921921    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921924    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921927    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921929    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921933    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921937    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921939    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921942    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921945    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921948    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921952    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921955    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921958    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.925226 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921961    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921964    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921967    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921970    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921973    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921977    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921981    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921985    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921989    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921993    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921996    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.921999    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922002    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922004    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922007    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.925856 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.922013    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922159    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922166    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922168    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922171    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922174    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922177    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922180    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922183    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922186    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922188    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922191    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922194    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922197    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922200    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922204    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922207    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922210    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922212    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922215    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:00.926279 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922219    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922224    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922228    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922232    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922235    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922238    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922242    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922245    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922248    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922252    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922255    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922257    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922260    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922262    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922265    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922267    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922270    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922273    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922275    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:00.926894 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922279    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922282    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922284    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922286    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922289    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922292    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922295    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922300    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922304    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922308    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922312    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922316    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922319    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922321    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922324    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922326    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922329    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922333    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922336    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:00.927602 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922340    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922343    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922345    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922348    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922352    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922354    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922357    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922360    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922362    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922365    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922368    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922370    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922375    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922380    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922385    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922388    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922391    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922393    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922397    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922400    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:00.928122 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922402    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922405    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922407    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922410    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922412    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922415    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922417    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922420    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922422    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922425    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922427    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922430    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922433    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922435    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922438    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922441    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922443    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922446    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922450    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922454    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:00.928868 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922459    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:00.922463    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.922468    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.922645    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.924339    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.925392    2564 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.925507    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.925562    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:00.929669 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.929005    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:00.931987 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.931966    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:00.940871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.940849    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:00.943857 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.943832    2564 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:00.944733 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.944711    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:00.948612 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.948580    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b31bce73-e2ea-494b-98d9-6b31e14e2af5:/dev/nvme0n1p3 cda44989-f48a-49a1-b2e0-a8dc90cb30d2:/dev/nvme0n1p4]
Sep 09 20:11:00.948728 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.948609    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:00.953967 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.953799    2564 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:00.952487222 +0000 UTC m=+0.327025307 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104509 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec294f3eaa898483da582405d04068ab SystemUUID:ec294f3e-aa89-8483-da58-2405d04068ab BootID:9c105f5e-69e7-4172-9311-8e21d85f15f5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:49:0d:db:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:49:0d:db:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:01:31:17:6e:d5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:00.953967 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.953948    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:00.954190 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.954154    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:00.957382 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.957352    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:00.957537 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.957387    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:00.958125 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.958114    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:00.958177 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.958127    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:00.958177 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.958136    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:00.958725 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.958714    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:00.959597 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.959584    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:00.959644 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.959615    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:00.962172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962160    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:00.962285 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962276    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:00.962315 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962306    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:00.962352 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962316    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:00.962352 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962336    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:00.962352 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962346    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:00.962442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962355    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:00.962839 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962826    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:00.962875 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.962853    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:00.965279 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.965252    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:00.969263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.969241    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:00.969350 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.969276    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:00.970606 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970590    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970612    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970618    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970623    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970632    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970637    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970651    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970656    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970663    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970669    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970678    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:00.970680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.970684    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:00.971314 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.971305    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:00.971314 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.971314    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:00.974124 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.974107    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:00.974383 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.974361    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:00.974442 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.974380    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:00.975304 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.975291    2564 server.go:1268] "Started kubelet"
Sep 09 20:11:00.975404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.975371    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:00.975485 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.975444    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:00.975525 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.975509    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:00.976787 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.976762    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:00.976998 ip-10-0-143-149 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:00.979052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.979037    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:00.979670 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.979632    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:00.981662 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.981647    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:00.982146 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982127    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:00.982345 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.982326    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:00.982711 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982698    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:00.982784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982767    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:00.982840 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982791    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:00.982891 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.981629    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18d3c0121a0d8e24  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-09-09 20:11:00.975259172 +0000 UTC m=+0.349797324,LastTimestamp:2026-09-09 20:11:00.975259172 +0000 UTC m=+0.349797324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Sep 09 20:11:00.982891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982890    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:00.982999 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.982898    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:00.983527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983511    2564 factory.go:55] Registering systemd factory
Sep 09 20:11:00.983527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983527    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:00.983784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983769    2564 factory.go:153] Registering CRI-O factory
Sep 09 20:11:00.983784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983786    2564 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:00.983924 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983842    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:00.983924 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983862    2564 factory.go:103] Registering Raw factory
Sep 09 20:11:00.983924 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.983876    2564 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:00.984313 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.984299    2564 manager.go:349] Starting recovery of all containers
Sep 09 20:11:00.989137 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.989106    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:11:00.989276 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:00.989202    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:11:00.995085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:00.995037    2564 manager.go:354] Recovery completed
Sep 09 20:11:01.001037 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.001017    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.004207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004180    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.004427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004226    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.004427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004251    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.004652 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004626    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gzlk5"
Sep 09 20:11:01.004965 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004950    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:01.005043 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004965    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:01.005043 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.004985    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:01.005043 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.005017    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.006349 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.006331    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.006420 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.006363    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.006420 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.006379    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.006420 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.006258    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18d3c0121bc73efb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-09-09 20:11:01.004205819 +0000 UTC m=+0.378743905,LastTimestamp:2026-09-09 20:11:01.004205819 +0000 UTC m=+0.378743905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Sep 09 20:11:01.007039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.006989    2564 policy_none.go:49] "None policy: Start"
Sep 09 20:11:01.007039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.007018    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:01.007039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.007034    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:01.008451 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.008438    2564 policy_none.go:47] "Start"
Sep 09 20:11:01.012436 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.012414    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gzlk5"
Sep 09 20:11:01.014438 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.014355    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-149.ec2.internal.18d3c0121bc7a8fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-149.ec2.internal,UID:ip-10-0-143-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-149.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-149.ec2.internal,},FirstTimestamp:2026-09-09 20:11:01.004232956 +0000 UTC m=+0.378771044,LastTimestamp:2026-09-09 20:11:01.004232956 +0000 UTC m=+0.378771044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-149.ec2.internal,}"
Sep 09 20:11:01.031892 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.031870    2564 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.031919    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.031933    2564 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.032285    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.032297    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.032435    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.032518    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.032529    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.033305    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:01.049011 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.033357    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.111158 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.111110    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:01.112480 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.112460    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:01.112639 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.112515    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:01.112639 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.112539    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:01.112639 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.112585    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:01.114955 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.114932    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:01.132940 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.132911    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.133966 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.133944    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.134088 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.133982    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.134088 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.133997    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.134088 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.134025    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.143153 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.143127    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.143262 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.143161    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-149.ec2.internal\": node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.166933 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.166835    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.212996 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.212957    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"]
Sep 09 20:11:01.213188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.213036    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.214170 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.214153    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.214245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.214184    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.214245 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.214195    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.215837 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.215824    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.215975 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.215954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.216033 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.215983    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.216603 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216586    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.216678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216620    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.216678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216631    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.216678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216591    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.216779 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216692    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.216779 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.216702    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.217772 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.217754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.217866 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.217782    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:01.218493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.218477    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:01.218577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.218509    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:01.218577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.218525    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:01.241912 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.241886    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-149.ec2.internal\" not found" node="ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.246413 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.246394    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-149.ec2.internal\" not found" node="ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.267516 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.267486    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.284470 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.284441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.284574 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.284473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.284574 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.284490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005ed891fe314d0a7d8b24d4cb67be1-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"9005ed891fe314d0a7d8b24d4cb67be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.367925 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.367891    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.385268 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.385268 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.385468 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005ed891fe314d0a7d8b24d4cb67be1-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"9005ed891fe314d0a7d8b24d4cb67be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.385468 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.385468 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385358    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36ba708629619c441ddda0b68c7e294c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal\" (UID: \"36ba708629619c441ddda0b68c7e294c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.385468 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.385384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005ed891fe314d0a7d8b24d4cb67be1-config\") pod \"kube-apiserver-proxy-ip-10-0-143-149.ec2.internal\" (UID: \"9005ed891fe314d0a7d8b24d4cb67be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.468288 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.468256    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.547762 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.547721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.550111 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.550089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.569191 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.569149    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.674253 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.674203    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.780464 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.780377    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.882642 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.882609    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-149.ec2.internal\" not found"
Sep 09 20:11:01.893189 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.893166    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:01.927470 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.927427    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:01.927990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.927602    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:01.927990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.927605    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:01.963591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.963549    2564 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:01.982540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.982509    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:01.983009 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.982994    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:01.983329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.983299    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:01.984421 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.984401    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw","openshift-image-registry/node-ca-vsxdf","openshift-multus/multus-fqlhz","openshift-network-diagnostics/network-check-target-mtz8b","openshift-ovn-kubernetes/ovnkube-node-q8td4","kube-system/global-pull-secret-syncer-rn6b9","kube-system/konnectivity-agent-g5zrk","openshift-cluster-node-tuning-operator/tuned-fdz7p","openshift-dns/node-resolver-gtncg","openshift-multus/multus-additional-cni-plugins-74l2q","openshift-multus/network-metrics-daemon-l82g9","openshift-network-operator/iptables-alerter-vfjzd"]
Sep 09 20:11:01.986465 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.986434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:01.988185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.988158    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:01.989399 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.989375    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tzbxf\""
Sep 09 20:11:01.989399 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.989393    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:01.989592 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.989428    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:01.989592 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.989458    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:01.989715 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.989694    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:01.989964 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.989948    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.990923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990600    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:01.990923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990651    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.990923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990674    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67rzl\""
Sep 09 20:11:01.990923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990770    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:01.990923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990904    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:01.991234 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.990954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:01.992110 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.992084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:01.993059 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993036    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:01.993282 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-87ssb\""
Sep 09 20:11:01.993445 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:01.993564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993462    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:01.993564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993474    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:01.993681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:01.993681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993431    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:01.993792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993688    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.993845 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.993827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:01.994360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.994345    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:01.994433 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.994361    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:01.994766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.994750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:01.996062 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.996047    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:01.996170 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.996118    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:01.996940 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.996906    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nkpvf\""
Sep 09 20:11:01.997113 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997096    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.997196 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:01.997196 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997159    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.997309 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45hvc\""
Sep 09 20:11:01.997367 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997323    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:01.997367 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbs67\""
Sep 09 20:11:01.997624 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.997608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:01.998980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.998961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:01.999097 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:01.999020    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:01.999718 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.999700    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:01.999881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.999858    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:01.999987 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:01.999895    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ckcc6\""
Sep 09 20:11:02.000533 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.000218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:02.000597 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.000433    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:02.000645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.000637    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"
Sep 09 20:11:02.000944 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.000923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.002343 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.002289    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal"]
Sep 09 20:11:02.002417 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.002368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.002876 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.002860    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:02.003151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.003136    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:02.003218 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.003148    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jqtj\""
Sep 09 20:11:02.004585 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.004568    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:02.004681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.004664    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:02.004844 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.004819    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlx9f\""
Sep 09 20:11:02.005018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.004985    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:02.013079 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.013042    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:02.013188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.013166    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal"]
Sep 09 20:11:02.016381 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.016331    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:01 +0000 UTC" deadline="2028-02-05 19:39:54.17463634 +0000 UTC"
Sep 09 20:11:02.016450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.016383    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12335h28m52.158257877s"
Sep 09 20:11:02.018300 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.018279    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-phdft"
Sep 09 20:11:02.028107 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.028081    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-phdft"
Sep 09 20:11:02.084215 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.084183    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:02.088522 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-systemd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.088671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-bin\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.088671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.088671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-iptables-alerter-script\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.088671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxd2\" (UniqueName: \"kubernetes.io/projected/a9ba811a-5b54-43e3-8324-0925702101a0-kube-api-access-9xxd2\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.088879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-k8s-cni-cncf-io\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.088879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-socket-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.088879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.088879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxsm\" (UniqueName: \"kubernetes.io/projected/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kube-api-access-znxsm\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.088879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-hostroot\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-etc-kubernetes\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088945    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovn-node-metrics-cert\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.088997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-script-lib\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-tmp-dir\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-sys-fs\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-multus-certs\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-slash\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-config\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-hosts-file\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.089172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz55n\" (UniqueName: \"kubernetes.io/projected/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-kube-api-access-nz55n\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-dbus\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ba811a-5b54-43e3-8324-0925702101a0-host\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cni-binary-copy\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-netns\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-conf\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-sys\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-system-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089416    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-kubelet\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-netns\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-var-lib-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-host\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-conf-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszkw\" (UniqueName: \"kubernetes.io/projected/ed289824-ea7f-446b-b9f7-b51a6b31a34d-kube-api-access-vszkw\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.089708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911f6be8-2fc6-40bb-9694-7e018bbb972a-agent-certs\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hwd\" (UniqueName: \"kubernetes.io/projected/d40242c9-05a1-4c5e-aa11-936ace80d2dd-kube-api-access-q5hwd\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-kubelet\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-daemon-config\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-netd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-kubelet-config\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-systemd\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-tmp\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-modprobe-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbcq\" (UniqueName: \"kubernetes.io/projected/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-kube-api-access-2fbcq\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-node-log\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911f6be8-2fc6-40bb-9694-7e018bbb972a-konnectivity-ca\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089890    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7w8\" (UniqueName: \"kubernetes.io/projected/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-kube-api-access-vb7w8\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-kubernetes\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.090369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-multus\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-etc-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.089995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4hn\" (UniqueName: \"kubernetes.io/projected/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-kube-api-access-5s4hn\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-systemd-units\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090126    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-env-overrides\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-run\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-var-lib-kubelet\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-tuned\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhl7\" (UniqueName: \"kubernetes.io/projected/c4bda37d-7d2d-441b-9e03-57bb934a62e1-kube-api-access-bzhl7\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ba811a-5b54-43e3-8324-0925702101a0-serviceca\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-bin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-device-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-system-cni-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.091000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cnibin\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-host-slash\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-log-socket\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysconfig\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-ovn\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-registration-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-lib-modules\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cnibin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-socket-dir-parent\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.091613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.090692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191057 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-kubelet\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-netns\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-kubelet\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-netns\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-var-lib-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-host\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.191281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-var-lib-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-conf-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszkw\" (UniqueName: \"kubernetes.io/projected/ed289824-ea7f-446b-b9f7-b51a6b31a34d-kube-api-access-vszkw\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-host\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-conf-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911f6be8-2fc6-40bb-9694-7e018bbb972a-agent-certs\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hwd\" (UniqueName: \"kubernetes.io/projected/d40242c9-05a1-4c5e-aa11-936ace80d2dd-kube-api-access-q5hwd\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-kubelet\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-daemon-config\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-netd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-kubelet-config\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-systemd\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.191595    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-tmp\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-modprobe-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.191634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-netd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-kubelet-config\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-kubelet\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-systemd\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.191708    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.69167541 +0000 UTC m=+2.066213498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191743    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbcq\" (UniqueName: \"kubernetes.io/projected/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-kube-api-access-2fbcq\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-node-log\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911f6be8-2fc6-40bb-9694-7e018bbb972a-konnectivity-ca\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7w8\" (UniqueName: \"kubernetes.io/projected/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-kube-api-access-vb7w8\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-kubernetes\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-multus\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-modprobe-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-etc-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4hn\" (UniqueName: \"kubernetes.io/projected/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-kube-api-access-5s4hn\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.192476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.191986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-systemd-units\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-env-overrides\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-kubernetes\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-run\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.192115    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-var-lib-kubelet\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-tuned\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.192189    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.692172627 +0000 UTC m=+2.066710715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhl7\" (UniqueName: \"kubernetes.io/projected/c4bda37d-7d2d-441b-9e03-57bb934a62e1-kube-api-access-bzhl7\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ba811a-5b54-43e3-8324-0925702101a0-serviceca\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-bin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-device-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192366    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-daemon-config\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-system-cni-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.193228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cnibin\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-host-slash\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-run\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911f6be8-2fc6-40bb-9694-7e018bbb972a-konnectivity-ca\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-log-socket\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-var-lib-kubelet\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysconfig\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-ovn\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-registration-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-lib-modules\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-systemd-units\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cnibin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192758    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-node-log\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-socket-dir-parent\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-multus\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-systemd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-bin\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-etc-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-iptables-alerter-script\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxd2\" (UniqueName: \"kubernetes.io/projected/a9ba811a-5b54-43e3-8324-0925702101a0-kube-api-access-9xxd2\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.192988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-k8s-cni-cncf-io\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193018    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-socket-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxsm\" (UniqueName: \"kubernetes.io/projected/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kube-api-access-znxsm\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193193    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-env-overrides\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-var-lib-cni-bin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-device-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.194896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-system-cni-dir\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d40242c9-05a1-4c5e-aa11-936ace80d2dd-cnibin\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193405    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cnibin\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193464    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-socket-dir-parent\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-log-socket\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193519    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-k8s-cni-cncf-io\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-openvswitch\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193598    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-systemd\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ba811a-5b54-43e3-8324-0925702101a0-serviceca\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-cni-bin\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysconfig\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-etc-selinux\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-run-ovn\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-registration-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-socket-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-lib-modules\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.195803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-hostroot\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-etc-kubernetes\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovn-node-metrics-cert\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-iptables-alerter-script\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-hostroot\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.193992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-script-lib\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-multus-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-host-slash\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-tmp-dir\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-etc-kubernetes\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-sys-fs\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-multus-certs\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-sys-fs\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-slash\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-config\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-multus-certs\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-hosts-file\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.196647 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-hosts-file\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz55n\" (UniqueName: \"kubernetes.io/projected/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-kube-api-access-nz55n\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed289824-ea7f-446b-b9f7-b51a6b31a34d-host-slash\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-dbus\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ba811a-5b54-43e3-8324-0925702101a0-host\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194430    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0a9382c-c5ae-4a35-8e07-09ea6026a485-dbus\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ba811a-5b54-43e3-8324-0925702101a0-host\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194431    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-tmp-dir\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-d\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194519    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-script-lib\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cni-binary-copy\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-netns\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-conf\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-sys\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-host-run-netns\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-system-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-system-cni-dir\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-sys\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.194779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-sysctl-conf\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.195018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovnkube-config\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.195021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-cni-binary-copy\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.196384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-etc-tuned\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.196411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4bda37d-7d2d-441b-9e03-57bb934a62e1-tmp\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.196831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911f6be8-2fc6-40bb-9694-7e018bbb972a-agent-certs\") pod \"konnectivity-agent-g5zrk\" (UID: \"911f6be8-2fc6-40bb-9694-7e018bbb972a\") " pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.197784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.197012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed289824-ea7f-446b-b9f7-b51a6b31a34d-ovn-node-metrics-cert\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.200873 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.200833    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:02.200873 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.200863    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:02.200873 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.200878    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.201226 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.200987    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.700931148 +0000 UTC m=+2.075469226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.201226 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.201120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszkw\" (UniqueName: \"kubernetes.io/projected/ed289824-ea7f-446b-b9f7-b51a6b31a34d-kube-api-access-vszkw\") pod \"ovnkube-node-q8td4\" (UID: \"ed289824-ea7f-446b-b9f7-b51a6b31a34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.201226 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.201198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hwd\" (UniqueName: \"kubernetes.io/projected/d40242c9-05a1-4c5e-aa11-936ace80d2dd-kube-api-access-q5hwd\") pod \"multus-additional-cni-plugins-74l2q\" (UID: \"d40242c9-05a1-4c5e-aa11-936ace80d2dd\") " pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.201895 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.201860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbcq\" (UniqueName: \"kubernetes.io/projected/a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491-kube-api-access-2fbcq\") pod \"multus-fqlhz\" (UID: \"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491\") " pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.202202 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.202179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7w8\" (UniqueName: \"kubernetes.io/projected/6a2a57af-fe8b-420d-8083-d08bcce8dcaa-kube-api-access-vb7w8\") pod \"node-resolver-gtncg\" (UID: \"6a2a57af-fe8b-420d-8083-d08bcce8dcaa\") " pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.202202 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.202222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxsm\" (UniqueName: \"kubernetes.io/projected/a9bc87eb-23ee-4b48-a85f-01a2a4dd311f-kube-api-access-znxsm\") pod \"aws-ebs-csi-driver-node-798gw\" (UID: \"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.203506 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.203485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhl7\" (UniqueName: \"kubernetes.io/projected/c4bda37d-7d2d-441b-9e03-57bb934a62e1-kube-api-access-bzhl7\") pod \"tuned-fdz7p\" (UID: \"c4bda37d-7d2d-441b-9e03-57bb934a62e1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.203890 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.203866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4hn\" (UniqueName: \"kubernetes.io/projected/99fec714-3a7f-47a4-8b2b-743bc4d1b60a-kube-api-access-5s4hn\") pod \"iptables-alerter-vfjzd\" (UID: \"99fec714-3a7f-47a4-8b2b-743bc4d1b60a\") " pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.204055 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.204037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz55n\" (UniqueName: \"kubernetes.io/projected/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-kube-api-access-nz55n\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.204322 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.204306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxd2\" (UniqueName: \"kubernetes.io/projected/a9ba811a-5b54-43e3-8324-0925702101a0-kube-api-access-9xxd2\") pod \"node-ca-vsxdf\" (UID: \"a9ba811a-5b54-43e3-8324-0925702101a0\") " pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.241691 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.241654    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ba708629619c441ddda0b68c7e294c.slice/crio-262614fed59ae29f70bde8eca750823fe50a862a7bfac0a7a0e2ee58fa5bdf4b WatchSource:0}: Error finding container 262614fed59ae29f70bde8eca750823fe50a862a7bfac0a7a0e2ee58fa5bdf4b: Status 404 returned error can't find the container with id 262614fed59ae29f70bde8eca750823fe50a862a7bfac0a7a0e2ee58fa5bdf4b
Sep 09 20:11:02.241985 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.241960    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9005ed891fe314d0a7d8b24d4cb67be1.slice/crio-3ad79da6595aad25ae2d98897728907f9626b1ad9aa8d4b4db1dd5bcd15f70b9 WatchSource:0}: Error finding container 3ad79da6595aad25ae2d98897728907f9626b1ad9aa8d4b4db1dd5bcd15f70b9: Status 404 returned error can't find the container with id 3ad79da6595aad25ae2d98897728907f9626b1ad9aa8d4b4db1dd5bcd15f70b9
Sep 09 20:11:02.247119 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.247099    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:02.310536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.310417    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vsxdf"
Sep 09 20:11:02.319468 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.319432    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ba811a_5b54_43e3_8324_0925702101a0.slice/crio-e4a43ce41050370fe7d71289bf66294c8ca12149d9f8232f5205419f34a930c5 WatchSource:0}: Error finding container e4a43ce41050370fe7d71289bf66294c8ca12149d9f8232f5205419f34a930c5: Status 404 returned error can't find the container with id e4a43ce41050370fe7d71289bf66294c8ca12149d9f8232f5205419f34a930c5
Sep 09 20:11:02.325409 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.325387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqlhz"
Sep 09 20:11:02.333582 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.333550    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b3ecd2_3ca5_4fbb_89ff_5bb610b05491.slice/crio-077401bb06097a7cd09ac8fc82ec8a9df0716850bdbf35c3e7c962d83a5bef58 WatchSource:0}: Error finding container 077401bb06097a7cd09ac8fc82ec8a9df0716850bdbf35c3e7c962d83a5bef58: Status 404 returned error can't find the container with id 077401bb06097a7cd09ac8fc82ec8a9df0716850bdbf35c3e7c962d83a5bef58
Sep 09 20:11:02.349535 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.349502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:02.351579 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.351455    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:02.357214 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.357185    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded289824_ea7f_446b_b9f7_b51a6b31a34d.slice/crio-30c39e0bb5649f594d266259349cca38d60afc8a91bfbfaa6f3c1da972ed8c39 WatchSource:0}: Error finding container 30c39e0bb5649f594d266259349cca38d60afc8a91bfbfaa6f3c1da972ed8c39: Status 404 returned error can't find the container with id 30c39e0bb5649f594d266259349cca38d60afc8a91bfbfaa6f3c1da972ed8c39
Sep 09 20:11:02.364705 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.364682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:02.372704 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.372667    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911f6be8_2fc6_40bb_9694_7e018bbb972a.slice/crio-00ab2e6360368ff7e9e05c19708736ff203f14ab81845924f3cc01c004f0aec6 WatchSource:0}: Error finding container 00ab2e6360368ff7e9e05c19708736ff203f14ab81845924f3cc01c004f0aec6: Status 404 returned error can't find the container with id 00ab2e6360368ff7e9e05c19708736ff203f14ab81845924f3cc01c004f0aec6
Sep 09 20:11:02.385277 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.385249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fdz7p"
Sep 09 20:11:02.393050 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.393019    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bda37d_7d2d_441b_9e03_57bb934a62e1.slice/crio-a5295f2cbcd227132e609f0ce3d10f17e802a2f19ac3eb62d054d772cbfefd1e WatchSource:0}: Error finding container a5295f2cbcd227132e609f0ce3d10f17e802a2f19ac3eb62d054d772cbfefd1e: Status 404 returned error can't find the container with id a5295f2cbcd227132e609f0ce3d10f17e802a2f19ac3eb62d054d772cbfefd1e
Sep 09 20:11:02.400924 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.400897    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtncg"
Sep 09 20:11:02.409456 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.409428    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2a57af_fe8b_420d_8083_d08bcce8dcaa.slice/crio-aa19b09ac227b6ef17ecaa5c737f7aafcd2467e6d328115970e7ceb74061b0e7 WatchSource:0}: Error finding container aa19b09ac227b6ef17ecaa5c737f7aafcd2467e6d328115970e7ceb74061b0e7: Status 404 returned error can't find the container with id aa19b09ac227b6ef17ecaa5c737f7aafcd2467e6d328115970e7ceb74061b0e7
Sep 09 20:11:02.418018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.417990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw"
Sep 09 20:11:02.421811 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.421790    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:02.427248 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.427222    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bc87eb_23ee_4b48_a85f_01a2a4dd311f.slice/crio-d5d886f57f3dc9136c1bbd5ec042cee981130de42f1ba25155440fd61e215309 WatchSource:0}: Error finding container d5d886f57f3dc9136c1bbd5ec042cee981130de42f1ba25155440fd61e215309: Status 404 returned error can't find the container with id d5d886f57f3dc9136c1bbd5ec042cee981130de42f1ba25155440fd61e215309
Sep 09 20:11:02.435008 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.434983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74l2q"
Sep 09 20:11:02.442427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.442403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vfjzd"
Sep 09 20:11:02.443527 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.443494    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40242c9_05a1_4c5e_aa11_936ace80d2dd.slice/crio-6a4b6f4c819abccfa9ac370a27ecfcf632db67b801aa70ce0ee88d8c516d5a50 WatchSource:0}: Error finding container 6a4b6f4c819abccfa9ac370a27ecfcf632db67b801aa70ce0ee88d8c516d5a50: Status 404 returned error can't find the container with id 6a4b6f4c819abccfa9ac370a27ecfcf632db67b801aa70ce0ee88d8c516d5a50
Sep 09 20:11:02.450452 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:11:02.450425    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fec714_3a7f_47a4_8b2b_743bc4d1b60a.slice/crio-b2b5fbb124ea072a70481397dd98569b3d85f8b2068525808423fed2e560c481 WatchSource:0}: Error finding container b2b5fbb124ea072a70481397dd98569b3d85f8b2068525808423fed2e560c481: Status 404 returned error can't find the container with id b2b5fbb124ea072a70481397dd98569b3d85f8b2068525808423fed2e560c481
Sep 09 20:11:02.697782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.697699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:02.697782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.697738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:02.697973 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.697844    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.697973 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.697853    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.697973 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.697904    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:03.697889508 +0000 UTC m=+3.072427596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.697973 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.697919    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:03.697912389 +0000 UTC m=+3.072450477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.798222 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.798179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:02.798413 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.798338    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:02.798413 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.798363    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:02.798413 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.798379    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.798572 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:02.798450    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:03.79842935 +0000 UTC m=+3.172967440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.862209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:02.862173    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:03.030519 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.030320    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:02 +0000 UTC" deadline="2028-04-25 23:56:21.373488282 +0000 UTC"
Sep 09 20:11:03.030519 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.030400    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14259h45m18.343092507s"
Sep 09 20:11:03.117564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.117528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:03.117763 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.117674    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:03.158256 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.158108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5zrk" event={"ID":"911f6be8-2fc6-40bb-9694-7e018bbb972a","Type":"ContainerStarted","Data":"00ab2e6360368ff7e9e05c19708736ff203f14ab81845924f3cc01c004f0aec6"}
Sep 09 20:11:03.175837 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.175687    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"30c39e0bb5649f594d266259349cca38d60afc8a91bfbfaa6f3c1da972ed8c39"}
Sep 09 20:11:03.188446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.188355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vsxdf" event={"ID":"a9ba811a-5b54-43e3-8324-0925702101a0","Type":"ContainerStarted","Data":"e4a43ce41050370fe7d71289bf66294c8ca12149d9f8232f5205419f34a930c5"}
Sep 09 20:11:03.219493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.219448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" event={"ID":"9005ed891fe314d0a7d8b24d4cb67be1","Type":"ContainerStarted","Data":"3ad79da6595aad25ae2d98897728907f9626b1ad9aa8d4b4db1dd5bcd15f70b9"}
Sep 09 20:11:03.248438 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.248358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"36ba708629619c441ddda0b68c7e294c","Type":"ContainerStarted","Data":"262614fed59ae29f70bde8eca750823fe50a862a7bfac0a7a0e2ee58fa5bdf4b"}
Sep 09 20:11:03.259288 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.259171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vfjzd" event={"ID":"99fec714-3a7f-47a4-8b2b-743bc4d1b60a","Type":"ContainerStarted","Data":"b2b5fbb124ea072a70481397dd98569b3d85f8b2068525808423fed2e560c481"}
Sep 09 20:11:03.280719 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.280622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw" event={"ID":"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f","Type":"ContainerStarted","Data":"d5d886f57f3dc9136c1bbd5ec042cee981130de42f1ba25155440fd61e215309"}
Sep 09 20:11:03.289492 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.289391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtncg" event={"ID":"6a2a57af-fe8b-420d-8083-d08bcce8dcaa","Type":"ContainerStarted","Data":"aa19b09ac227b6ef17ecaa5c737f7aafcd2467e6d328115970e7ceb74061b0e7"}
Sep 09 20:11:03.301957 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.301915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fdz7p" event={"ID":"c4bda37d-7d2d-441b-9e03-57bb934a62e1","Type":"ContainerStarted","Data":"a5295f2cbcd227132e609f0ce3d10f17e802a2f19ac3eb62d054d772cbfefd1e"}
Sep 09 20:11:03.316558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.316515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqlhz" event={"ID":"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491","Type":"ContainerStarted","Data":"077401bb06097a7cd09ac8fc82ec8a9df0716850bdbf35c3e7c962d83a5bef58"}
Sep 09 20:11:03.337969 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.337926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerStarted","Data":"6a4b6f4c819abccfa9ac370a27ecfcf632db67b801aa70ce0ee88d8c516d5a50"}
Sep 09 20:11:03.705771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.705684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:03.705965 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.705784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:03.705965 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.705898    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:03.705965 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.705961    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:05.705940999 +0000 UTC m=+5.080479090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:03.706448 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.706426    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:03.706550 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.706490    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:05.706472776 +0000 UTC m=+5.081010867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:03.807863 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:03.807158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:03.807863 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.807384    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:03.807863 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.807408    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:03.807863 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.807422    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:03.807863 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:03.807485    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:05.807466793 +0000 UTC m=+5.182004883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:04.030805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:04.030754    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:02 +0000 UTC" deadline="2028-04-10 09:58:54.549928093 +0000 UTC"
Sep 09 20:11:04.030805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:04.030804    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13885h47m50.519129643s"
Sep 09 20:11:04.113988 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:04.113424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:04.113988 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:04.113580    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:04.113988 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:04.113771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:04.113988 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:04.113936    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:04.140067 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:04.138896    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:05.116383 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:05.115808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:05.116383 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.115948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:05.722674 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:05.722631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:05.722880 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:05.722700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:05.722880 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.722833    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:05.723097 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.722900    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.722879477 +0000 UTC m=+9.097417567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:05.723404 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.723382    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:05.723494 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.723447    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.723430236 +0000 UTC m=+9.097968312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:05.823600 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:05.823553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:05.823793 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.823738    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:05.823793 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.823764    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:05.823793 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.823778    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:05.823977 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:05.823845    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.823824604 +0000 UTC m=+9.198362683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:06.113794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:06.112953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:06.113794 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:06.113132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:06.113794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:06.113631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:06.113794 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:06.113738    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:07.112990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:07.112951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:07.113499 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:07.113110    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:08.113236 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:08.113200    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:08.113236 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:08.113235    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:08.113761 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:08.113355    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:08.113761 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:08.113496    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:09.113299 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:09.113263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:09.113763 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.113416    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:09.756328    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:09.756390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.756546    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.756611    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.7565937 +0000 UTC m=+17.131131786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.757003    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.757099 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.757051    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.757034882 +0000 UTC m=+17.131572970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.856985 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:09.856941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:09.857169 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.857118    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:09.857169 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.857137    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:09.857169 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.857150    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:09.857303 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:09.857215    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.857197059 +0000 UTC m=+17.231735146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:10.113642 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:10.113606    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:10.114286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:10.113649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:10.114286 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:10.113763    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:10.114286 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:10.113881    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:11.114148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:11.114117    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:11.114559 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:11.114249    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:12.113368 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:12.113333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:12.113368 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:12.113370    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:12.113601 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:12.113474    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:12.113640 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:12.113611    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:13.113114 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:13.113077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:13.113577 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:13.113200    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:14.113237 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:14.113197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:14.113693 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:14.113197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:14.113693 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:14.113312    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:14.113693 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:14.113385    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:15.113113 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:15.113077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:15.113297 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:15.113212    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:16.113570 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:16.113535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:16.114039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:16.113539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:16.114039 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:16.113675    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:16.114039 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:16.113739    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:17.113224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:17.113176    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:17.113402 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.113310    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:17.820736 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:17.820700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:17.820736 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:17.820744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:17.821173 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.820838    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.821173 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.820846    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.821173 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.820889    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.820875545 +0000 UTC m=+33.195413630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.821173 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.820901    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.820895853 +0000 UTC m=+33.195433924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.921861 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:17.921823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:17.922041 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.921970    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:17.922041 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.921989    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:17.922041 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.921998    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:17.922192 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:17.922049    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.922037202 +0000 UTC m=+33.296575273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:18.113089 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:18.113040    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:18.113089 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:18.113086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:18.113312 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:18.113173    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:18.113376 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:18.113340    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:19.113182 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:19.113145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:19.113719 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:19.113278    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:20.113494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:20.113453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:20.113944 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:20.113600    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:20.113944 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:20.113642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:20.113944 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:20.113717    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:21.113897 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:21.113848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:21.114394 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:21.114016    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:22.113439 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.112855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:22.113543 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:22.113504    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:22.113543 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.112882    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:22.113621 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:22.113591    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:22.395794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.395768    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396173    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed289824-ea7f-446b-b9f7-b51a6b31a34d" containerID="7323d6251eaf18430b2764cc1fdf89567c1e40ff5412b0bd0a1c27fd64dd52cd" exitCode=1
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396241    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"a50071fa0078b676aa6e52c30ac9b4e5f82366c281566a985e60b412f40c806d"}
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"043ee28cbd19bbe70ea7c64cf5df1328199923c6fb3b35e4321f11c3f0149eac"}
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"97a3a507373c2ac1d9af64115e7c38d703f227503c7b214653cdb561132dab13"}
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"3c20fc24eea13c95a9c63eb92e016bde1fe08b8f5386cb9af5dc272b63d0fbd6"}
Sep 09 20:11:22.396483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396446    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerDied","Data":"7323d6251eaf18430b2764cc1fdf89567c1e40ff5412b0bd0a1c27fd64dd52cd"}
Sep 09 20:11:22.396717 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.396494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"ea2ded6cb2479d0aaa9862c55f153a620e16a10bc6b256ec7f2db4f7762587f8"}
Sep 09 20:11:22.397610 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.397586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vsxdf" event={"ID":"a9ba811a-5b54-43e3-8324-0925702101a0","Type":"ContainerStarted","Data":"cd4ea592cae2c7be1dffa0d45d47693fe5b0b0815ae1360b15a3e47e065385bf"}
Sep 09 20:11:22.399111 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.399085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" event={"ID":"9005ed891fe314d0a7d8b24d4cb67be1","Type":"ContainerStarted","Data":"83bd7236d8622b39ccfcd1b8f36bc11bdb4dad26393877ca83ff48ba0387f10f"}
Sep 09 20:11:22.400769 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.400745    2564 generic.go:356] "Generic (PLEG): container finished" podID="36ba708629619c441ddda0b68c7e294c" containerID="7e9770aea8493f1c6a15c4d4a5c36b098a5cd2ff9eec9f80f087d13f3b71a951" exitCode=0
Sep 09 20:11:22.400854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.400821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"36ba708629619c441ddda0b68c7e294c","Type":"ContainerDied","Data":"7e9770aea8493f1c6a15c4d4a5c36b098a5cd2ff9eec9f80f087d13f3b71a951"}
Sep 09 20:11:22.402418 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.402398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw" event={"ID":"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f","Type":"ContainerStarted","Data":"6afacd5ab10caf238c214099ecc7984ae1bc5b7c57a2bdcd44fa9f756c92820e"}
Sep 09 20:11:22.403769 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.403748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtncg" event={"ID":"6a2a57af-fe8b-420d-8083-d08bcce8dcaa","Type":"ContainerStarted","Data":"feddf27600a55a36b0394aaabce17173dbe657ebc3ba280d12851f9f4847c9ef"}
Sep 09 20:11:22.405030 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.405011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fdz7p" event={"ID":"c4bda37d-7d2d-441b-9e03-57bb934a62e1","Type":"ContainerStarted","Data":"724676d6e0bf8a01b73144878bf9bf6070b564528172a0c6a0c2206d3906699c"}
Sep 09 20:11:22.406450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.406430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqlhz" event={"ID":"a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491","Type":"ContainerStarted","Data":"85e140c8431f524b4083e040336296edb355424468925e76726c04482b2bafab"}
Sep 09 20:11:22.407906 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.407885    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="ee93b65625520258c544081d30ce3beba183351ebadb95dfb361d484beed1688" exitCode=0
Sep 09 20:11:22.407981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.407964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"ee93b65625520258c544081d30ce3beba183351ebadb95dfb361d484beed1688"}
Sep 09 20:11:22.409375 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.409323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g5zrk" event={"ID":"911f6be8-2fc6-40bb-9694-7e018bbb972a","Type":"ContainerStarted","Data":"94247b755ffb83b2456a19d35026d36b7fb5960479fa42a16348e764d1386cef"}
Sep 09 20:11:22.412876 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.412836    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vsxdf" podStartSLOduration=6.152099364 podStartE2EDuration="21.4128232s" totalImagesPullingTime="15.260723836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.412187177 +0000 UTC m=+21.786725271" watchObservedRunningTime="2026-09-09 20:11:22.4128232 +0000 UTC m=+21.787361293"
Sep 09 20:11:22.429804 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.429763    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fqlhz" podStartSLOduration=1.8917654069999998 podStartE2EDuration="21.429749248s" totalImagesPullingTime="19.537983841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.429681419 +0000 UTC m=+21.804219512" watchObservedRunningTime="2026-09-09 20:11:22.429749248 +0000 UTC m=+21.804287342"
Sep 09 20:11:22.448523 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.448474    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gtncg" podStartSLOduration=2.298919253 podStartE2EDuration="21.448458065s" totalImagesPullingTime="19.149538812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.448094385 +0000 UTC m=+21.822632475" watchObservedRunningTime="2026-09-09 20:11:22.448458065 +0000 UTC m=+21.822996159"
Sep 09 20:11:22.502631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.502583    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-149.ec2.internal" podStartSLOduration=20.502565036 podStartE2EDuration="20.502565036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.502184429 +0000 UTC m=+21.876722522" watchObservedRunningTime="2026-09-09 20:11:22.502565036 +0000 UTC m=+21.877103131"
Sep 09 20:11:22.517735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.517692    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g5zrk" podStartSLOduration=2.332548411 podStartE2EDuration="21.517670907s" totalImagesPullingTime="19.185122496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.517548318 +0000 UTC m=+21.892086416" watchObservedRunningTime="2026-09-09 20:11:22.517670907 +0000 UTC m=+21.892209002"
Sep 09 20:11:22.534660 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:22.534589    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fdz7p" podStartSLOduration=2.351787216 podStartE2EDuration="21.534570767s" totalImagesPullingTime="19.182783551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.534170548 +0000 UTC m=+21.908708642" watchObservedRunningTime="2026-09-09 20:11:22.534570767 +0000 UTC m=+21.909108860"
Sep 09 20:11:23.090970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.090737    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:23.091371 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.091348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:23.113969 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.113937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:23.114152 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:23.114095    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:23.412915 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.412826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" event={"ID":"36ba708629619c441ddda0b68c7e294c","Type":"ContainerStarted","Data":"8eca52ebf5df7e212230a46635f8af856df5b637ba9a10ea4494a84579a04c65"}
Sep 09 20:11:23.414251 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.414220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vfjzd" event={"ID":"99fec714-3a7f-47a4-8b2b-743bc4d1b60a","Type":"ContainerStarted","Data":"64135694a355e182b3d08254e335753984ac8771002d32aed9406a9a77013846"}
Sep 09 20:11:23.428164 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.428103    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-149.ec2.internal" podStartSLOduration=21.428082384 podStartE2EDuration="21.428082384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:23.427498099 +0000 UTC m=+22.802036195" watchObservedRunningTime="2026-09-09 20:11:23.428082384 +0000 UTC m=+22.802620479"
Sep 09 20:11:23.443106 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.443032    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vfjzd" podStartSLOduration=2.334376566 podStartE2EDuration="21.443017427s" totalImagesPullingTime="19.108640861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:23.442834046 +0000 UTC m=+22.817372140" watchObservedRunningTime="2026-09-09 20:11:23.443017427 +0000 UTC m=+22.817555550"
Sep 09 20:11:23.763898 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:23.763871    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:24.043185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.043054    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:23.763891599Z","UUID":"6534432b-f355-4e62-86ce-4074d3fb10bd","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:24.044874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.044852    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:24.045006 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.044888    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:24.113363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.113334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:24.113564 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:24.113463    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:24.113564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.113503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:24.113695 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:24.113584    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:24.420657 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.420618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw" event={"ID":"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f","Type":"ContainerStarted","Data":"ebe0d60ff9ea621cf6c6973e27479a1faa1282681250b7edb9298e7d45b2128b"}
Sep 09 20:11:24.420657 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:24.420669    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:11:25.112784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.112755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:25.112932 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:25.112896    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:25.425227 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.425138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw" event={"ID":"a9bc87eb-23ee-4b48-a85f-01a2a4dd311f","Type":"ContainerStarted","Data":"7f141fcf628456078bd91d6a864c0e286cdf6e65e7f838f40fd5f3cddf76c4c1"}
Sep 09 20:11:25.428678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.428649    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:11:25.429091 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.429045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"b0444ad196eec65877aab0167d99c2ff678ab0a212313f7ee1e128b92c460f43"}
Sep 09 20:11:25.442028 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.441981    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-798gw" podStartSLOduration=1.671446526 podStartE2EDuration="23.441960634s" totalImagesPullingTime="21.770514108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:25.440900224 +0000 UTC m=+24.815438318" watchObservedRunningTime="2026-09-09 20:11:25.441960634 +0000 UTC m=+24.816498727"
Sep 09 20:11:25.667886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.667843    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:25.668082 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.668042    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:11:25.668578 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:25.668548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g5zrk"
Sep 09 20:11:26.113296 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:26.113259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:26.113508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:26.113263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:26.113508 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:26.113398    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:26.113508 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:26.113458    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:27.113437 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.113248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:27.113822 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:27.113518    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:27.435427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.435392    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="0fe4e08cb9b17b72f441e1d12590b795fe049519f40d95ddc2b5cd3ed508f4b8" exitCode=0
Sep 09 20:11:27.435603 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.435477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"0fe4e08cb9b17b72f441e1d12590b795fe049519f40d95ddc2b5cd3ed508f4b8"}
Sep 09 20:11:27.439259 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.439239    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:11:27.439803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.439777    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"56515a5a727d89f56d583ce930a64e9811a1e623cfcc6c29aff7b7087fe74fe2"}
Sep 09 20:11:27.440073 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.440038    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:27.440179 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.440137    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:27.440266 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.440251    2564 scope.go:117] "RemoveContainer" containerID="7323d6251eaf18430b2764cc1fdf89567c1e40ff5412b0bd0a1c27fd64dd52cd"
Sep 09 20:11:27.455944 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:27.455923    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:28.113286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.113185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:28.113578 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:28.113335    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:28.114231 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.113727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:28.114231 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:28.113823    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:28.446199 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.446125    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:11:28.446525 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.446500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" event={"ID":"ed289824-ea7f-446b-b9f7-b51a6b31a34d","Type":"ContainerStarted","Data":"8e13bf1d7c8e3b7499d357691714561c88fc76a2c82be906e850dac1331ef62f"}
Sep 09 20:11:28.446705 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.446686    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:28.462543 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.462074    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:11:28.482248 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.482190    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4" podStartSLOduration=8.230405915 podStartE2EDuration="27.482166511s" totalImagesPullingTime="19.251760596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.48127404 +0000 UTC m=+27.855812137" watchObservedRunningTime="2026-09-09 20:11:28.482166511 +0000 UTC m=+27.856704609"
Sep 09 20:11:28.841847 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.841819    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rn6b9"]
Sep 09 20:11:28.842038 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.841918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:28.842038 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:28.842007    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:28.845022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.844994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l82g9"]
Sep 09 20:11:28.845191 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.845122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:28.845257 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:28.845236    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:28.845663 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.845642    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mtz8b"]
Sep 09 20:11:28.845752 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:28.845726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:28.845819 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:28.845800    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:29.451264 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:29.451230    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="335b7e3b82377a053d87afb5279798de1424184cd6be933707604e3e79701422" exitCode=0
Sep 09 20:11:29.451965 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:29.451312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"335b7e3b82377a053d87afb5279798de1424184cd6be933707604e3e79701422"}
Sep 09 20:11:30.113638 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:30.113517    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:30.113638 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:30.113518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:30.113888 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:30.113640    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:30.113888 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:30.113716    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:31.114672 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:31.114475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:31.115042 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:31.114733    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:31.458829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:31.458743    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="d06dcb63f66837e340f3e426945f4ccfc4c6f346cb5d6bda340ff3c4e250f450" exitCode=0
Sep 09 20:11:31.458829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:31.458791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"d06dcb63f66837e340f3e426945f4ccfc4c6f346cb5d6bda340ff3c4e250f450"}
Sep 09 20:11:32.113614 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:32.113580    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:32.113793 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:32.113580    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:32.113793 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:32.113741    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:32.113881 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:32.113805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:33.112851 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:33.112811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:33.113353 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.112945    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz8b" podUID="18ab09d5-8034-465e-a2fd-f53a5ed05581"
Sep 09 20:11:33.840948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:33.840913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:33.841170 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:33.840967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:33.841170 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.841083    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:33.841170 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.841083    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:33.841315 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.841170    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret podName:c0a9382c-c5ae-4a35-8e07-09ea6026a485 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.841149887 +0000 UTC m=+65.215687974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret") pod "global-pull-secret-syncer-rn6b9" (UID: "c0a9382c-c5ae-4a35-8e07-09ea6026a485") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:33.841315 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.841243    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs podName:e2feec63-e0b9-4f53-ac1c-b4e4c876c9df nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.841228504 +0000 UTC m=+65.215766579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs") pod "network-metrics-daemon-l82g9" (UID: "e2feec63-e0b9-4f53-ac1c-b4e4c876c9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:33.941764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:33.941726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:33.941994 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.941884    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:33.941994 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.941907    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:33.941994 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.941919    2564 projected.go:196] Error preparing data for projected volume kube-api-access-llx9p for pod openshift-network-diagnostics/network-check-target-mtz8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:33.941994 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:33.941985    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p podName:18ab09d5-8034-465e-a2fd-f53a5ed05581 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.941966874 +0000 UTC m=+65.316504948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-llx9p" (UniqueName: "kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p") pod "network-check-target-mtz8b" (UID: "18ab09d5-8034-465e-a2fd-f53a5ed05581") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:34.113364 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.113280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:34.113920 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:34.113416    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l82g9" podUID="e2feec63-e0b9-4f53-ac1c-b4e4c876c9df"
Sep 09 20:11:34.113920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.113453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:34.113920 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:34.113550    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rn6b9" podUID="c0a9382c-c5ae-4a35-8e07-09ea6026a485"
Sep 09 20:11:34.893114 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.893086    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-149.ec2.internal" event="NodeReady"
Sep 09 20:11:34.893312 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.893244    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:34.933312 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.933228    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:11:34.960263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.960234    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"]
Sep 09 20:11:34.960864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.960830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:34.966194 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.965865    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:11:34.966375 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.966160    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:11:34.966613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.966197    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:11:34.966771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.966342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4lhjh\""
Sep 09 20:11:34.970973 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.970950    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:11:34.977685 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.977569    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"]
Sep 09 20:11:34.978097 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.977703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"
Sep 09 20:11:34.981575 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.981550    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:34.981704 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.981587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:34.981766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:34.981596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dns2d\""
Sep 09 20:11:35.005816 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.005789    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"]
Sep 09 20:11:35.005984 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.005943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.007968 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.007943    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:11:35.008240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.008208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.008240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.008227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zl6tf\""
Sep 09 20:11:35.008416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.008208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.026051 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.026014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lmvxs"]
Sep 09 20:11:35.026051 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.026042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.028370 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.028345    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:11:35.028508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.028378    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-26knj\""
Sep 09 20:11:35.028508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.028400    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.028508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.028378    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.028508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.028351    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:11:35.039824 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.039800    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"]
Sep 09 20:11:35.039946 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.039932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.042797 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.042775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.042941 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.042813    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:11:35.043016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.042943    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vbxqd\""
Sep 09 20:11:35.043016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.042969    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.043166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.043048    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:11:35.047836 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.047457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:11:35.048783 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.048758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.048902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.048819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.048902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.048866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.049013 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.048904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvg4\" (UniqueName: \"kubernetes.io/projected/032fd8cb-9219-4537-944d-9352add4873f-kube-api-access-mqvg4\") pod \"volume-data-source-validator-695c895bd6-49tn2\" (UID: \"032fd8cb-9219-4537-944d-9352add4873f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"
Sep 09 20:11:35.049092 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.049011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.049092 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.049053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.049219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.049104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrzk\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.049219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.049123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.049219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.049142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.054207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.054183    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"]
Sep 09 20:11:35.054317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.054298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.056483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.056466    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:11:35.056591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.056521    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.056746 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.056525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:11:35.056852 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.056755    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wtpmx\""
Sep 09 20:11:35.056913 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.056849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.076486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:11:35.076486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076494    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"]
Sep 09 20:11:35.076684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076520    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"]
Sep 09 20:11:35.076684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076537    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"]
Sep 09 20:11:35.076684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076557    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"]
Sep 09 20:11:35.076684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076582    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kpld5"]
Sep 09 20:11:35.076865 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.076750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.079323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.079294    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q6mpl\""
Sep 09 20:11:35.079457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.079387    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:11:35.079532 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.079309    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.080022 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.079999    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.080226 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.080206    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:11:35.096139 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.096107    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nh9z8"]
Sep 09 20:11:35.096277 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.096232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.098816 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.098751    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zh2q\""
Sep 09 20:11:35.098816 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.098809    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:35.099008 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.098820    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:35.116739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.116709    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.119119 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.119092    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.119270 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.119102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:11:35.119270 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.119184    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:11:35.119270 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.119213    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t6hxh\""
Sep 09 20:11:35.119832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.119671    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.120892 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.120874    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mnn2h"]
Sep 09 20:11:35.124447 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.124427    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:11:35.136714 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.136692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-87c784c5b-k6grx"]
Sep 09 20:11:35.136848 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.136823    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h"
Sep 09 20:11:35.139220 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.139191    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.139333 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.139236    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zt52p\""
Sep 09 20:11:35.139517 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.139490    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.150243 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.150378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.150378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c883-e15c-40ee-90db-19caf5a8b6ea-config-volume\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3affe817-470e-426a-aeae-1e75a35602dc-config\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.150422    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.150439    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150491    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"]
Sep 09 20:11:35.150521 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.150521    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.650498051 +0000 UTC m=+35.025036139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150531    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mbn2f"]
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/974ebb76-2a57-4c95-971c-6f6be973a3d9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150600    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdea4a8f-b525-4646-b6dd-39da4ba10271-serving-cert\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvg4\" (UniqueName: \"kubernetes.io/projected/032fd8cb-9219-4537-944d-9352add4873f-kube-api-access-mqvg4\") pod \"volume-data-source-validator-695c895bd6-49tn2\" (UID: \"032fd8cb-9219-4537-944d-9352add4873f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3affe817-470e-426a-aeae-1e75a35602dc-serving-cert\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-config\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150760    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.150871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.150821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrzk\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78r8\" (UniqueName: \"kubernetes.io/projected/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-kube-api-access-j78r8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cb2\" (UniqueName: \"kubernetes.io/projected/974ebb76-2a57-4c95-971c-6f6be973a3d9-kube-api-access-c5cb2\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8544c883-e15c-40ee-90db-19caf5a8b6ea-tmp-dir\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-trusted-ca\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9kd\" (UniqueName: \"kubernetes.io/projected/3affe817-470e-426a-aeae-1e75a35602dc-kube-api-access-kx9kd\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfjl\" (UniqueName: \"kubernetes.io/projected/cdea4a8f-b525-4646-b6dd-39da4ba10271-kube-api-access-2vfjl\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzwb\" (UniqueName: \"kubernetes.io/projected/8544c883-e15c-40ee-90db-19caf5a8b6ea-kube-api-access-4rzwb\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq9k\" (UniqueName: \"kubernetes.io/projected/d70fa5fc-4c59-48d9-895c-591e860629f3-kube-api-access-8pq9k\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.151990 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.151857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.152831 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.152195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.152896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.152880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.154205 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:11:35.154495 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154469    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:11:35.154495 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:11:35.154633 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154623    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vn26m\""
Sep 09 20:11:35.154806 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154786    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.154921 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154904    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.155016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.154904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:11:35.156740 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.156716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.156836 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.156743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.160771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.160744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvg4\" (UniqueName: \"kubernetes.io/projected/032fd8cb-9219-4537-944d-9352add4873f-kube-api-access-mqvg4\") pod \"volume-data-source-validator-695c895bd6-49tn2\" (UID: \"032fd8cb-9219-4537-944d-9352add4873f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"
Sep 09 20:11:35.161550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.161525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.161881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.161858    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrzk\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.166732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166662    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lmvxs"]
Sep 09 20:11:35.166732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166702    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mnn2h"]
Sep 09 20:11:35.166732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166717    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nh9z8"]
Sep 09 20:11:35.166732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpld5"]
Sep 09 20:11:35.166962 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166756    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mbn2f"]
Sep 09 20:11:35.166962 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-87c784c5b-k6grx"]
Sep 09 20:11:35.166962 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.166802    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-s6c5t"]
Sep 09 20:11:35.167485 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.167462    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.167922 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.167903    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:11:35.169761 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.169734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xhs4j\""
Sep 09 20:11:35.169967 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.169935    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52lgp\""
Sep 09 20:11:35.169967 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.169957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:35.170149 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.170045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:35.170283 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.170261    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:35.189090 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.189049    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-s6c5t"]
Sep 09 20:11:35.189247 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.189226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.192133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.192089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 20:11:35.192238 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.192135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 20:11:35.192238 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.192209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-scvms\""
Sep 09 20:11:35.253256 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8544c883-e15c-40ee-90db-19caf5a8b6ea-tmp-dir\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-tmp\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-default-certificate\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253324    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78r8\" (UniqueName: \"kubernetes.io/projected/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-kube-api-access-j78r8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb8161c5-4de2-42be-b02e-c73ac0a5c015-nginx-conf\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9kd\" (UniqueName: \"kubernetes.io/projected/3affe817-470e-426a-aeae-1e75a35602dc-kube-api-access-kx9kd\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.253442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-snapshots\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdea4a8f-b525-4646-b6dd-39da4ba10271-serving-cert\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22nc\" (UniqueName: \"kubernetes.io/projected/f749172f-1796-42aa-9d8e-29c96523be1e-kube-api-access-l22nc\") pod \"network-check-source-685456655f-mnn2h\" (UID: \"f749172f-1796-42aa-9d8e-29c96523be1e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8544c883-e15c-40ee-90db-19caf5a8b6ea-tmp-dir\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cb2\" (UniqueName: \"kubernetes.io/projected/974ebb76-2a57-4c95-971c-6f6be973a3d9-kube-api-access-c5cb2\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfjl\" (UniqueName: \"kubernetes.io/projected/cdea4a8f-b525-4646-b6dd-39da4ba10271-kube-api-access-2vfjl\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-stats-auth\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45pgk\" (UniqueName: \"kubernetes.io/projected/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-kube-api-access-45pgk\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.253814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhgs\" (UniqueName: \"kubernetes.io/projected/aba3eb39-9f90-4766-be49-f459acbf5379-kube-api-access-fnhgs\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzwb\" (UniqueName: \"kubernetes.io/projected/8544c883-e15c-40ee-90db-19caf5a8b6ea-kube-api-access-4rzwb\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b713f385-521f-48a1-8d5e-2fc44e380c56-cert\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253885    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3eb39-9f90-4766-be49-f459acbf5379-serving-cert\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-trusted-ca\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq9k\" (UniqueName: \"kubernetes.io/projected/d70fa5fc-4c59-48d9-895c-591e860629f3-kube-api-access-8pq9k\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.253973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4n9\" (UniqueName: \"kubernetes.io/projected/b713f385-521f-48a1-8d5e-2fc44e380c56-kube-api-access-hz4n9\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c883-e15c-40ee-90db-19caf5a8b6ea-config-volume\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3affe817-470e-426a-aeae-1e75a35602dc-config\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/974ebb76-2a57-4c95-971c-6f6be973a3d9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3affe817-470e-426a-aeae-1e75a35602dc-serving-cert\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.254416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-config\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.255087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254918    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8544c883-e15c-40ee-90db-19caf5a8b6ea-config-volume\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.255087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.254936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-config\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.255386 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255362    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:35.255502 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255397    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:35.255502 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255437    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.75541811 +0000 UTC m=+35.129956198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:35.255502 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.255432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/974ebb76-2a57-4c95-971c-6f6be973a3d9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.255502 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255464    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.755446484 +0000 UTC m=+35.129984578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:35.255739 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255541    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:35.255739 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.255579    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.755567957 +0000 UTC m=+35.130106036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:35.255846 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.255810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdea4a8f-b525-4646-b6dd-39da4ba10271-trusted-ca\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.255932 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.255913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3affe817-470e-426a-aeae-1e75a35602dc-config\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.256266 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.256247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.257395 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.257373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3affe817-470e-426a-aeae-1e75a35602dc-serving-cert\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.257626 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.257603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.257740 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.257606    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdea4a8f-b525-4646-b6dd-39da4ba10271-serving-cert\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.262577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.262514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9kd\" (UniqueName: \"kubernetes.io/projected/3affe817-470e-426a-aeae-1e75a35602dc-kube-api-access-kx9kd\") pod \"service-ca-operator-859f94c4b-x8nlz\" (UID: \"3affe817-470e-426a-aeae-1e75a35602dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.264579 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.264541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfjl\" (UniqueName: \"kubernetes.io/projected/cdea4a8f-b525-4646-b6dd-39da4ba10271-kube-api-access-2vfjl\") pod \"console-operator-85b454874b-lmvxs\" (UID: \"cdea4a8f-b525-4646-b6dd-39da4ba10271\") " pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.265018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.264992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78r8\" (UniqueName: \"kubernetes.io/projected/8e5dfd22-9ef7-488b-9819-5eb4fbba4035-kube-api-access-j78r8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dcg2g\" (UID: \"8e5dfd22-9ef7-488b-9819-5eb4fbba4035\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.265219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.265196    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzwb\" (UniqueName: \"kubernetes.io/projected/8544c883-e15c-40ee-90db-19caf5a8b6ea-kube-api-access-4rzwb\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.265546 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.265526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq9k\" (UniqueName: \"kubernetes.io/projected/d70fa5fc-4c59-48d9-895c-591e860629f3-kube-api-access-8pq9k\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.265864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.265840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cb2\" (UniqueName: \"kubernetes.io/projected/974ebb76-2a57-4c95-971c-6f6be973a3d9-kube-api-access-c5cb2\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.293198 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.293167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"
Sep 09 20:11:35.335310 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.335278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"
Sep 09 20:11:35.350535 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.350498    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:35.354789 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.354756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.354989 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.354912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22nc\" (UniqueName: \"kubernetes.io/projected/f749172f-1796-42aa-9d8e-29c96523be1e-kube-api-access-l22nc\") pod \"network-check-source-685456655f-mnn2h\" (UID: \"f749172f-1796-42aa-9d8e-29c96523be1e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h"
Sep 09 20:11:35.354989 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.354927    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:35.354989 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.354949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.354992    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.854971633 +0000 UTC m=+35.229509720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-stats-auth\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45pgk\" (UniqueName: \"kubernetes.io/projected/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-kube-api-access-45pgk\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhgs\" (UniqueName: \"kubernetes.io/projected/aba3eb39-9f90-4766-be49-f459acbf5379-kube-api-access-fnhgs\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b713f385-521f-48a1-8d5e-2fc44e380c56-cert\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.355200    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.855185331 +0000 UTC m=+35.229723422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3eb39-9f90-4766-be49-f459acbf5379-serving-cert\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4n9\" (UniqueName: \"kubernetes.io/projected/b713f385-521f-48a1-8d5e-2fc44e380c56-kube-api-access-hz4n9\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-tmp\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-default-certificate\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb8161c5-4de2-42be-b02e-c73ac0a5c015-nginx-conf\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.355475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-snapshots\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355907 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.355907 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.355758    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-tmp\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.356239 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.356145    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:35.356239 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.356175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aba3eb39-9f90-4766-be49-f459acbf5379-snapshots\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.356239 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.356215    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:35.856199572 +0000 UTC m=+35.230737657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:35.356545 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.356489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb8161c5-4de2-42be-b02e-c73ac0a5c015-nginx-conf\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.356545 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.356525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aba3eb39-9f90-4766-be49-f459acbf5379-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.358325 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.358297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aba3eb39-9f90-4766-be49-f459acbf5379-serving-cert\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.358594 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.358571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-stats-auth\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.358637 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.358584    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b713f385-521f-48a1-8d5e-2fc44e380c56-cert\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.358695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.358678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-default-certificate\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.364043 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.364017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"
Sep 09 20:11:35.365207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.365185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22nc\" (UniqueName: \"kubernetes.io/projected/f749172f-1796-42aa-9d8e-29c96523be1e-kube-api-access-l22nc\") pod \"network-check-source-685456655f-mnn2h\" (UID: \"f749172f-1796-42aa-9d8e-29c96523be1e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h"
Sep 09 20:11:35.365914 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.365883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhgs\" (UniqueName: \"kubernetes.io/projected/aba3eb39-9f90-4766-be49-f459acbf5379-kube-api-access-fnhgs\") pod \"insights-operator-76bcd6ccf5-nh9z8\" (UID: \"aba3eb39-9f90-4766-be49-f459acbf5379\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.365981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.365883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45pgk\" (UniqueName: \"kubernetes.io/projected/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-kube-api-access-45pgk\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.366248 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.366228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4n9\" (UniqueName: \"kubernetes.io/projected/b713f385-521f-48a1-8d5e-2fc44e380c56-kube-api-access-hz4n9\") pod \"ingress-canary-mbn2f\" (UID: \"b713f385-521f-48a1-8d5e-2fc44e380c56\") " pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.427762 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.427730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8"
Sep 09 20:11:35.447100 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.447015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h"
Sep 09 20:11:35.484579 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.484547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mbn2f"
Sep 09 20:11:35.659571 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.659145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:35.659571 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.659262    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:35.659571 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.659415    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:35.659571 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.659479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.659458291 +0000 UTC m=+36.033996384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:35.760705 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.760669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:35.760834 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.760748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:35.760834 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.760771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:35.760919 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.760856    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:35.760919 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.760870    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:35.760979 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.760930    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.760913847 +0000 UTC m=+36.135451919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:35.760979 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.760965    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.760949836 +0000 UTC m=+36.135487959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:35.760979 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.760972    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:35.761111 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.761011    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.761003754 +0000 UTC m=+36.135541831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:35.861915 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.861875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.861915 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.861922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:35.862136 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.861991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:35.862136 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.862028    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:35.862136 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.862119    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.862102692 +0000 UTC m=+36.236640777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:35.862136 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.862120    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:35.862268 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.862158    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.862145336 +0000 UTC m=+36.236683429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:35.862268 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:35.862181    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.862172297 +0000 UTC m=+36.236710376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:35.926612 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.926577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g"]
Sep 09 20:11:35.936701 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.936664    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mbn2f"]
Sep 09 20:11:35.936778 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.936719    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz"]
Sep 09 20:11:35.942197 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.942176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-mnn2h"]
Sep 09 20:11:35.943577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.943314    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nh9z8"]
Sep 09 20:11:35.944280 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.944230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2"]
Sep 09 20:11:35.946394 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:35.945390    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-lmvxs"]
Sep 09 20:11:36.112866 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.112839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:11:36.112998 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.112849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:11:36.115569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.115546    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:36.115569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.115553    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dttj\""
Sep 09 20:11:36.115730 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.115582    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:36.474350 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.474261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mbn2f" event={"ID":"b713f385-521f-48a1-8d5e-2fc44e380c56","Type":"ContainerStarted","Data":"cba388578dee32405af9c5b27eb4f86b067df647116fa75b5c3bff6bc7ea45e8"}
Sep 09 20:11:36.477156 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.477085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz" event={"ID":"3affe817-470e-426a-aeae-1e75a35602dc","Type":"ContainerStarted","Data":"2bd33fc36d9e15ea8ddb4585011df23f1a2d0302f2e27870c27af8f5ae721138"}
Sep 09 20:11:36.478812 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.478743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h" event={"ID":"f749172f-1796-42aa-9d8e-29c96523be1e","Type":"ContainerStarted","Data":"021f37d442dd581035a212043cc468eaff2a6cfc6dc3753e47f75bf104a62e2c"}
Sep 09 20:11:36.481769 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.481708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8" event={"ID":"aba3eb39-9f90-4766-be49-f459acbf5379","Type":"ContainerStarted","Data":"9c8b055ca7f65b5c27cbe14a888d51d2a10f4909476353a87a251741028844a1"}
Sep 09 20:11:36.483308 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.483262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" event={"ID":"cdea4a8f-b525-4646-b6dd-39da4ba10271","Type":"ContainerStarted","Data":"a730319af71b8443c203d2acde46072acc798f871b5d7b2ac0a118d4b0310d88"}
Sep 09 20:11:36.484893 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.484860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g" event={"ID":"8e5dfd22-9ef7-488b-9819-5eb4fbba4035","Type":"ContainerStarted","Data":"c5675cb6d1782fce0016b5bc88403bdce8a031b31f5187843fdc3ca6a49e73df"}
Sep 09 20:11:36.486318 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.486295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2" event={"ID":"032fd8cb-9219-4537-944d-9352add4873f","Type":"ContainerStarted","Data":"1a29b7a32cc542b5c054e4972fdc6b6232927e8a54dc793c0f6eb719641dc61e"}
Sep 09 20:11:36.490263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.490236    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="4e748644006954d3f131a3fd72600362a0e7ba1851bec1ae274ba504a8cb9838" exitCode=0
Sep 09 20:11:36.490345 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.490281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"4e748644006954d3f131a3fd72600362a0e7ba1851bec1ae274ba504a8cb9838"}
Sep 09 20:11:36.672564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.672156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:36.672564 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.672483    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:36.672564 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.672501    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:36.673222 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.672860    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.672835468 +0000 UTC m=+38.047373540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.773429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.773570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.773663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.773786    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.773852    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.773831965 +0000 UTC m=+38.148370040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.773917    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.773950    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.773939734 +0000 UTC m=+38.148477813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.774008    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:36.774412 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.774036    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.774026718 +0000 UTC m=+38.148564794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:36.875247 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.875208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:36.875409 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.875327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:36.875409 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:36.875361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:36.875551 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.875489    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:36.875608 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.875561    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.875543136 +0000 UTC m=+38.250081223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:36.876151 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.875969    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.875952801 +0000 UTC m=+38.250490877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:36.876151 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.876052    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:36.876151 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:36.876107    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:38.87609432 +0000 UTC m=+38.250632395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:37.507124 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:37.506880    2564 generic.go:356] "Generic (PLEG): container finished" podID="d40242c9-05a1-4c5e-aa11-936ace80d2dd" containerID="a232dd769f8fd0a703be71e83db10812c71dc87e6032b06c4fbde9c6a24773ee" exitCode=0
Sep 09 20:11:37.507124 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:37.506943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerDied","Data":"a232dd769f8fd0a703be71e83db10812c71dc87e6032b06c4fbde9c6a24773ee"}
Sep 09 20:11:38.512724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.512457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2" event={"ID":"032fd8cb-9219-4537-944d-9352add4873f","Type":"ContainerStarted","Data":"8b7451c040da4cd3c506a3939c6b16f524912a1f237ad4ab6807fea1c3774e5e"}
Sep 09 20:11:38.518976 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.518522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74l2q" event={"ID":"d40242c9-05a1-4c5e-aa11-936ace80d2dd","Type":"ContainerStarted","Data":"6398dd40dcbe00ded2d13570e8e474e7e95711ae601ec84d94c917558ef02800"}
Sep 09 20:11:38.528381 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.528147    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-49tn2" podStartSLOduration=8.089979507 podStartE2EDuration="9.528124026s" totalImagesPullingTime="1.438144519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:38.527150015 +0000 UTC m=+37.901688110" watchObservedRunningTime="2026-09-09 20:11:38.528124026 +0000 UTC m=+37.902662121"
Sep 09 20:11:38.548184 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.547691    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-74l2q" podStartSLOduration=6.415215761 podStartE2EDuration="36.547671209s" totalImagesPullingTime="30.132455448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:38.545562602 +0000 UTC m=+37.920100697" watchObservedRunningTime="2026-09-09 20:11:38.547671209 +0000 UTC m=+37.922209304"
Sep 09 20:11:38.695091 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.694986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:38.695262 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.695159    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:38.695262 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.695175    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:38.695262 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.695233    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.695216136 +0000 UTC m=+42.069754223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:38.795897 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.795859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:38.796087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.795992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:38.796087 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796024    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:38.796199 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796119    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.796099458 +0000 UTC m=+42.170637548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:38.796199 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796143    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:38.796199 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.796180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:38.796356 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796208    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.796188183 +0000 UTC m=+42.170726284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:38.796356 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796266    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:38.796356 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.796305    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.796291239 +0000 UTC m=+42.170829335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:38.897650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.897601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:38.897833 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.897745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:38.897833 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.897779    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.897758528 +0000 UTC m=+42.272296613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:38.897960 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.897838    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:38.897960 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.897888    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.897873099 +0000 UTC m=+42.272411191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:38.897960 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:38.897831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:38.898141 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.897898    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:38.898141 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:38.898010    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.897997114 +0000 UTC m=+42.272535199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:42.736326 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.736278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:42.736735 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.736453    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:42.736735 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.736478    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:42.736735 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.736550    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.736527266 +0000 UTC m=+50.111065353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:42.837549 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.837509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:42.837549 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.837557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:42.837764 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837691    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:42.837797 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.837763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:42.837797 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837789    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:42.837870 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837843    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.837824043 +0000 UTC m=+50.212362117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:42.837915 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837872    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.837861467 +0000 UTC m=+50.212399548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:42.837915 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837883    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:42.837976 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.837928    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.837915374 +0000 UTC m=+50.212453474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:42.939292 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.939253    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:42.939486 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.939406    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:42.939486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.939418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:42.939601 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.939492    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.939471101 +0000 UTC m=+50.314009190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:42.939601 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.939544    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:42.939601 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.939600    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.939583794 +0000 UTC m=+50.314121896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:42.939747 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:42.939646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:42.939747 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:42.939739    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:50.939725129 +0000 UTC m=+50.314263226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:44.538595 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.538558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mbn2f" event={"ID":"b713f385-521f-48a1-8d5e-2fc44e380c56","Type":"ContainerStarted","Data":"34c294860ff4c5e9ecf205aa14ad4634851f5542c5b88cf60abfeb660b48d130"}
Sep 09 20:11:44.540920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.540501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz" event={"ID":"3affe817-470e-426a-aeae-1e75a35602dc","Type":"ContainerStarted","Data":"89fd2a0ed43423f1181e6d94ea15a70289dd5c749f15d31825e0f9e845a509f3"}
Sep 09 20:11:44.548543 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.548511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h" event={"ID":"f749172f-1796-42aa-9d8e-29c96523be1e","Type":"ContainerStarted","Data":"46208ef97d6e28166cc1180321a9058ad35e7671496f4d879284c17b29f5867e"}
Sep 09 20:11:44.550285 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.550256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8" event={"ID":"aba3eb39-9f90-4766-be49-f459acbf5379","Type":"ContainerStarted","Data":"8aa79faa851e0cbf74d6da373f7bbb4b7ddb7281dfc5a325fd32ca3e6ec9c9d2"}
Sep 09 20:11:44.552020 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.552001    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/0.log"
Sep 09 20:11:44.552173 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.552040    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdea4a8f-b525-4646-b6dd-39da4ba10271" containerID="8be541fa9d4deb6e713929d294c744d36e2f035f3c57b61fec240dabdaef7947" exitCode=255
Sep 09 20:11:44.552173 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.552088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" event={"ID":"cdea4a8f-b525-4646-b6dd-39da4ba10271","Type":"ContainerDied","Data":"8be541fa9d4deb6e713929d294c744d36e2f035f3c57b61fec240dabdaef7947"}
Sep 09 20:11:44.552345 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.552323    2564 scope.go:117] "RemoveContainer" containerID="8be541fa9d4deb6e713929d294c744d36e2f035f3c57b61fec240dabdaef7947"
Sep 09 20:11:44.554539 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.554494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g" event={"ID":"8e5dfd22-9ef7-488b-9819-5eb4fbba4035","Type":"ContainerStarted","Data":"84ad69dc0c97dcbf0ff69494de4bc52a837b3911f4204cf49ad93a12cbb62db2"}
Sep 09 20:11:44.555743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.555706    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mbn2f" podStartSLOduration=2.907629011 podStartE2EDuration="10.555692141s" totalImagesPullingTime="7.64806313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.554509623 +0000 UTC m=+43.929047731" watchObservedRunningTime="2026-09-09 20:11:44.555692141 +0000 UTC m=+43.930230235"
Sep 09 20:11:44.569817 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.569157    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-mnn2h" podStartSLOduration=7.912495267 podStartE2EDuration="15.569120087s" totalImagesPullingTime="7.65662482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.568565784 +0000 UTC m=+43.943103879" watchObservedRunningTime="2026-09-09 20:11:44.569120087 +0000 UTC m=+43.943658185"
Sep 09 20:11:44.583205 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.583125    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz" podStartSLOduration=7.923934548 podStartE2EDuration="15.583105313s" totalImagesPullingTime="7.659170765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.582118106 +0000 UTC m=+43.956656197" watchObservedRunningTime="2026-09-09 20:11:44.583105313 +0000 UTC m=+43.957643408"
Sep 09 20:11:44.602121 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.602024    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8" podStartSLOduration=7.953632 podStartE2EDuration="15.602002767s" totalImagesPullingTime="7.648370767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.600030565 +0000 UTC m=+43.974568654" watchObservedRunningTime="2026-09-09 20:11:44.602002767 +0000 UTC m=+43.976540863"
Sep 09 20:11:44.643365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.642709    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g" podStartSLOduration=7.99229269 podStartE2EDuration="15.642684332s" totalImagesPullingTime="7.650391642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.641484677 +0000 UTC m=+44.016022771" watchObservedRunningTime="2026-09-09 20:11:44.642684332 +0000 UTC m=+44.017222426"
Sep 09 20:11:44.898411 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.898325    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nw29n"]
Sep 09 20:11:44.914129 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.914096    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nw29n"]
Sep 09 20:11:44.914312 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.914165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:44.917527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.916138    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:11:44.917527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.916484    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:11:44.917527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:44.917381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dnt6\""
Sep 09 20:11:45.062052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.062010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-crio-socket\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.062323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.062150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.062323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.062186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5czk\" (UniqueName: \"kubernetes.io/projected/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-api-access-j5czk\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.062323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.062307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-data-volume\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.062476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.062398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.162928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-data-volume\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163267 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-crio-socket\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163267 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.163153    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.163267 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.163223    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:45.663207157 +0000 UTC m=+45.037745264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.163267 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.163247    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.163462 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.163298    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:45.663282761 +0000 UTC m=+45.037820846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.163462 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163462 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5czk\" (UniqueName: \"kubernetes.io/projected/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-api-access-j5czk\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163462 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163350    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-crio-socket\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.163462 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.163406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-data-volume\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.172213 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.172183    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5czk\" (UniqueName: \"kubernetes.io/projected/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-api-access-j5czk\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.350760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.350723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:45.350760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.350772    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:45.559914 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.559875    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/1.log"
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.560529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/0.log"
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.560564    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdea4a8f-b525-4646-b6dd-39da4ba10271" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab" exitCode=255
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.561394    2564 scope.go:117] "RemoveContainer" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.561548    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.561593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" event={"ID":"cdea4a8f-b525-4646-b6dd-39da4ba10271","Type":"ContainerDied","Data":"be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"}
Sep 09 20:11:45.561792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.561650    2564 scope.go:117] "RemoveContainer" containerID="8be541fa9d4deb6e713929d294c744d36e2f035f3c57b61fec240dabdaef7947"
Sep 09 20:11:45.668566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.668523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.668761 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.668642    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.668761 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.668706    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:46.668687478 +0000 UTC m=+46.043225567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.669509 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:45.669346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:45.669509 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.669483    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.669679 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:45.669534    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:46.669519375 +0000 UTC m=+46.044057465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:46.570150 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:46.570122    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/1.log"
Sep 09 20:11:46.570507 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:46.570460    2564 scope.go:117] "RemoveContainer" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"
Sep 09 20:11:46.570649 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:46.570629    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:11:46.677607 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:46.677567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:46.677779 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:46.677664    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:46.677840 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:46.677805    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.677783512 +0000 UTC m=+48.052321594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:46.677901 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:46.677855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:46.677975 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:46.677959    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:46.678050 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:46.678037    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.678020041 +0000 UTC m=+48.052558123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:47.075841 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:47.075814    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtncg_6a2a57af-fe8b-420d-8083-d08bcce8dcaa/dns-node-resolver/0.log"
Sep 09 20:11:47.574545 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:47.574515    2564 scope.go:117] "RemoveContainer" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"
Sep 09 20:11:47.574928 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:47.574685    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:11:48.076174 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:48.076148    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vsxdf_a9ba811a-5b54-43e3-8324-0925702101a0/node-ca/0.log"
Sep 09 20:11:48.696073 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:48.696025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:48.696484 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:48.696156    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:48.696484 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:48.696159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:48.696484 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:48.696235    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:52.696215994 +0000 UTC m=+52.070754089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:48.696484 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:48.696309    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:48.696484 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:48.696376    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:52.696357748 +0000 UTC m=+52.070895835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:48.876588 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:48.876551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mbn2f_b713f385-521f-48a1-8d5e-2fc44e380c56/serve-healthcheck-canary/0.log"
Sep 09 20:11:50.814221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:50.814186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:11:50.814680 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.814334    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:50.814680 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.814350    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59679977db-fltqh: secret "image-registry-tls" not found
Sep 09 20:11:50.814680 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.814409    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls podName:bfe26766-68fa-4503-b62e-bcf304742a08 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.814389591 +0000 UTC m=+66.188927674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls") pod "image-registry-59679977db-fltqh" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08") : secret "image-registry-tls" not found
Sep 09 20:11:50.915411 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:50.915363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:11:50.915611 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:50.915459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:11:50.915611 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:50.915483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:11:50.915611 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915520    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:50.915611 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915585    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls podName:8544c883-e15c-40ee-90db-19caf5a8b6ea nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.915569833 +0000 UTC m=+66.290107930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls") pod "dns-default-kpld5" (UID: "8544c883-e15c-40ee-90db-19caf5a8b6ea") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:50.915611 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915601    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:50.915810 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915608    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:11:50.915810 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915641    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls podName:974ebb76-2a57-4c95-971c-6f6be973a3d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.915630125 +0000 UTC m=+66.290168197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9zrpr" (UID: "974ebb76-2a57-4c95-971c-6f6be973a3d9") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:11:50.915810 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:50.915661    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls podName:d70fa5fc-4c59-48d9-895c-591e860629f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.915646175 +0000 UTC m=+66.290184247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mgs55" (UID: "d70fa5fc-4c59-48d9-895c-591e860629f3") : secret "samples-operator-tls" not found
Sep 09 20:11:51.015934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:51.015896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:11:51.016137 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:51.016078    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:11:51.016181 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:51.016141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:51.016181 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:51.016156    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert podName:fb8161c5-4de2-42be-b02e-c73ac0a5c015 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.016136344 +0000 UTC m=+66.390674430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-s6c5t" (UID: "fb8161c5-4de2-42be-b02e-c73ac0a5c015") : secret "networking-console-plugin-cert" not found
Sep 09 20:11:51.016292 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:51.016226    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.016215818 +0000 UTC m=+66.390753907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : configmap references non-existent config key: service-ca.crt
Sep 09 20:11:51.016292 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:51.016280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:11:51.016392 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:51.016364    2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:11:51.016439 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:51.016401    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs podName:5f74bf6b-25f7-439d-98c7-1b79b2e89c82 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.016389967 +0000 UTC m=+66.390928055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs") pod "router-default-87c784c5b-k6grx" (UID: "5f74bf6b-25f7-439d-98c7-1b79b2e89c82") : secret "router-metrics-certs-default" not found
Sep 09 20:11:52.732425 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:52.732381    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:52.732806 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:52.732472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:52.732806 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:52.732565    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:52.732806 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:52.732626    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls podName:c7d1bd14-c0ae-4f9f-b680-f63734dfce35 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:00.732613036 +0000 UTC m=+60.107151121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nw29n" (UID: "c7d1bd14-c0ae-4f9f-b680-f63734dfce35") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:52.732964 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:52.732938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:11:55.350931 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.350890    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:55.351428 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.350988    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:11:55.351428 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.351284    2564 scope.go:117] "RemoveContainer" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"
Sep 09 20:11:55.603618 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.603538    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:11:55.604010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.603992    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/1.log"
Sep 09 20:11:55.604069 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.604032    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdea4a8f-b525-4646-b6dd-39da4ba10271" containerID="c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702" exitCode=255
Sep 09 20:11:55.604118 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.604103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" event={"ID":"cdea4a8f-b525-4646-b6dd-39da4ba10271","Type":"ContainerDied","Data":"c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702"}
Sep 09 20:11:55.604188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.604148    2564 scope.go:117] "RemoveContainer" containerID="be0aa5428ec8598497a26332dda2f35ba36820a67dd6c1d9a004ca5e7b11c1ab"
Sep 09 20:11:55.604372 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:55.604353    2564 scope.go:117] "RemoveContainer" containerID="c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702"
Sep 09 20:11:55.604572 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:55.604554    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:11:56.613731 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:56.613701    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:11:56.614133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:11:56.614019    2564 scope.go:117] "RemoveContainer" containerID="c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702"
Sep 09 20:11:56.614200 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:11:56.614181    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:12:00.464181 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:00.464151    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q8td4"
Sep 09 20:12:00.802599 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:00.802560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:12:00.804843 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:00.804823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d1bd14-c0ae-4f9f-b680-f63734dfce35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nw29n\" (UID: \"c7d1bd14-c0ae-4f9f-b680-f63734dfce35\") " pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:12:00.827013 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:00.826975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nw29n"
Sep 09 20:12:01.097358 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:01.097279    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nw29n"]
Sep 09 20:12:01.100855 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:01.100829    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d1bd14_c0ae_4f9f_b680_f63734dfce35.slice/crio-7342aa376f89f2ad03c900f12d4bfe697fe9c55f6ec2db660278617a4494c8b5 WatchSource:0}: Error finding container 7342aa376f89f2ad03c900f12d4bfe697fe9c55f6ec2db660278617a4494c8b5: Status 404 returned error can't find the container with id 7342aa376f89f2ad03c900f12d4bfe697fe9c55f6ec2db660278617a4494c8b5
Sep 09 20:12:01.629886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:01.629850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nw29n" event={"ID":"c7d1bd14-c0ae-4f9f-b680-f63734dfce35","Type":"ContainerStarted","Data":"d9b160a61bb5b352064f5a9a0043878be6950c81ec9a524c806487cb50e6dfb9"}
Sep 09 20:12:01.630298 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:01.629908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nw29n" event={"ID":"c7d1bd14-c0ae-4f9f-b680-f63734dfce35","Type":"ContainerStarted","Data":"7342aa376f89f2ad03c900f12d4bfe697fe9c55f6ec2db660278617a4494c8b5"}
Sep 09 20:12:02.635827 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:02.635790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nw29n" event={"ID":"c7d1bd14-c0ae-4f9f-b680-f63734dfce35","Type":"ContainerStarted","Data":"ebf7459290c6aab3a808e07c941945b2a940cba29c485ad77a31c1993610ab93"}
Sep 09 20:12:03.640826 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:03.640788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nw29n" event={"ID":"c7d1bd14-c0ae-4f9f-b680-f63734dfce35","Type":"ContainerStarted","Data":"aeb12364c8bdf1512b19fa796a7584c564697b3bd2e01d36e3c5523965b61081"}
Sep 09 20:12:03.657489 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:03.657437    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nw29n" podStartSLOduration=17.629940388 podStartE2EDuration="19.657419583s" totalImagesPullingTime="2.027479195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:03.655837816 +0000 UTC m=+63.030375923" watchObservedRunningTime="2026-09-09 20:12:03.657419583 +0000 UTC m=+63.031957676"
Sep 09 20:12:05.350829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.350778    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:12:05.351224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.350851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:12:05.351224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.351183    2564 scope.go:117] "RemoveContainer" containerID="c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702"
Sep 09 20:12:05.351350 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:05.351331    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-lmvxs_openshift-console-operator(cdea4a8f-b525-4646-b6dd-39da4ba10271)\"" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podUID="cdea4a8f-b525-4646-b6dd-39da4ba10271"
Sep 09 20:12:05.848002 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.847957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:12:05.848229 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.848131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:12:05.850433 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.850409    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:05.850552 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.850442    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:05.861360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.861329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0a9382c-c5ae-4a35-8e07-09ea6026a485-original-pull-secret\") pod \"global-pull-secret-syncer-rn6b9\" (UID: \"c0a9382c-c5ae-4a35-8e07-09ea6026a485\") " pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:12:05.861463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.861425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2feec63-e0b9-4f53-ac1c-b4e4c876c9df-metrics-certs\") pod \"network-metrics-daemon-l82g9\" (UID: \"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df\") " pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:12:05.949028 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.948989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:12:05.951509 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:05.951478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llx9p\" (UniqueName: \"kubernetes.io/projected/18ab09d5-8034-465e-a2fd-f53a5ed05581-kube-api-access-llx9p\") pod \"network-check-target-mtz8b\" (UID: \"18ab09d5-8034-465e-a2fd-f53a5ed05581\") " pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:12:06.093197 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.093163    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52lgp\""
Sep 09 20:12:06.101963 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.101894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:12:06.151154 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.151121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rn6b9"
Sep 09 20:12:06.159496 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.159272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dttj\""
Sep 09 20:12:06.174121 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.170079    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l82g9"
Sep 09 20:12:06.394611 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.394531    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mtz8b"]
Sep 09 20:12:06.397791 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:06.397763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ab09d5_8034_465e_a2fd_f53a5ed05581.slice/crio-04656b6775de361a6076023fd9771866b8f523207205e9c7d128ab7b9a09bc46 WatchSource:0}: Error finding container 04656b6775de361a6076023fd9771866b8f523207205e9c7d128ab7b9a09bc46: Status 404 returned error can't find the container with id 04656b6775de361a6076023fd9771866b8f523207205e9c7d128ab7b9a09bc46
Sep 09 20:12:06.443627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.443476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rn6b9"]
Sep 09 20:12:06.446391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.446369    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l82g9"]
Sep 09 20:12:06.447600 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:06.447408    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a9382c_c5ae_4a35_8e07_09ea6026a485.slice/crio-573bd08aba59a63e4a4a049f876f2e8faebde2376583f5e04ed51f42234e322e WatchSource:0}: Error finding container 573bd08aba59a63e4a4a049f876f2e8faebde2376583f5e04ed51f42234e322e: Status 404 returned error can't find the container with id 573bd08aba59a63e4a4a049f876f2e8faebde2376583f5e04ed51f42234e322e
Sep 09 20:12:06.449443 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:06.449421    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2feec63_e0b9_4f53_ac1c_b4e4c876c9df.slice/crio-467e907c0b4291d404bbe54e785d40835eb164e8e0e83b3e8624c0bd65e0e305 WatchSource:0}: Error finding container 467e907c0b4291d404bbe54e785d40835eb164e8e0e83b3e8624c0bd65e0e305: Status 404 returned error can't find the container with id 467e907c0b4291d404bbe54e785d40835eb164e8e0e83b3e8624c0bd65e0e305
Sep 09 20:12:06.650459 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.650370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rn6b9" event={"ID":"c0a9382c-c5ae-4a35-8e07-09ea6026a485","Type":"ContainerStarted","Data":"573bd08aba59a63e4a4a049f876f2e8faebde2376583f5e04ed51f42234e322e"}
Sep 09 20:12:06.651478 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.651445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l82g9" event={"ID":"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df","Type":"ContainerStarted","Data":"467e907c0b4291d404bbe54e785d40835eb164e8e0e83b3e8624c0bd65e0e305"}
Sep 09 20:12:06.652774 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.652753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mtz8b" event={"ID":"18ab09d5-8034-465e-a2fd-f53a5ed05581","Type":"ContainerStarted","Data":"4a09d2d0030dac0abe107f659ac06a41deb75f1fd490f89558f2b4d76fc1b758"}
Sep 09 20:12:06.652867 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.652792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mtz8b" event={"ID":"18ab09d5-8034-465e-a2fd-f53a5ed05581","Type":"ContainerStarted","Data":"04656b6775de361a6076023fd9771866b8f523207205e9c7d128ab7b9a09bc46"}
Sep 09 20:12:06.652909 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.652894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:12:06.668326 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.668283    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mtz8b" podStartSLOduration=65.668269293 podStartE2EDuration="1m5.668269293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:06.667824801 +0000 UTC m=+66.042362895" watchObservedRunningTime="2026-09-09 20:12:06.668269293 +0000 UTC m=+66.042807387"
Sep 09 20:12:06.862859 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.862822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:06.865551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.865517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"image-registry-59679977db-fltqh\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") " pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:06.963764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.963670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:12:06.963764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.963718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:12:06.963973 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.963782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:12:06.966385 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.966361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8544c883-e15c-40ee-90db-19caf5a8b6ea-metrics-tls\") pod \"dns-default-kpld5\" (UID: \"8544c883-e15c-40ee-90db-19caf5a8b6ea\") " pod="openshift-dns/dns-default-kpld5"
Sep 09 20:12:06.966707 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.966682    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/974ebb76-2a57-4c95-971c-6f6be973a3d9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9zrpr\" (UID: \"974ebb76-2a57-4c95-971c-6f6be973a3d9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:12:06.966781 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:06.966693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d70fa5fc-4c59-48d9-895c-591e860629f3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mgs55\" (UID: \"d70fa5fc-4c59-48d9-895c-591e860629f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:12:07.065124 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.065088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:07.065333 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.065146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:12:07.065333 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.065203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:07.065827 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.065799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-service-ca-bundle\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:07.067881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.067853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f74bf6b-25f7-439d-98c7-1b79b2e89c82-metrics-certs\") pod \"router-default-87c784c5b-k6grx\" (UID: \"5f74bf6b-25f7-439d-98c7-1b79b2e89c82\") " pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:07.067986 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.067951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb8161c5-4de2-42be-b02e-c73ac0a5c015-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-s6c5t\" (UID: \"fb8161c5-4de2-42be-b02e-c73ac0a5c015\") " pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:12:07.081627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.081599    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4lhjh\""
Sep 09 20:12:07.089443 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.089418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:07.119365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.119335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zl6tf\""
Sep 09 20:12:07.127319 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.127276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"
Sep 09 20:12:07.192051 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.192022    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q6mpl\""
Sep 09 20:12:07.200996 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.200754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"
Sep 09 20:12:07.209182 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.209159    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zh2q\""
Sep 09 20:12:07.217196 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.217135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpld5"
Sep 09 20:12:07.280345 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.280272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vn26m\""
Sep 09 20:12:07.288311 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.288247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:07.302166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.302144    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-scvms\""
Sep 09 20:12:07.310766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.310738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t"
Sep 09 20:12:07.376793 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.376709    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:12:07.380199 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:07.380167    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe26766_68fa_4503_b62e_bcf304742a08.slice/crio-499c512a15f331efff780940e005944690ce3a9c6aee86c1fbea872b564c12f0 WatchSource:0}: Error finding container 499c512a15f331efff780940e005944690ce3a9c6aee86c1fbea872b564c12f0: Status 404 returned error can't find the container with id 499c512a15f331efff780940e005944690ce3a9c6aee86c1fbea872b564c12f0
Sep 09 20:12:07.411243 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.411218    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55"]
Sep 09 20:12:07.484191 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.483883    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr"]
Sep 09 20:12:07.487334 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:07.487289    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974ebb76_2a57_4c95_971c_6f6be973a3d9.slice/crio-0c9fa7a2a2c4ab1b40ea08986a431f486496a7105ecc49aff964bc63746e42fb WatchSource:0}: Error finding container 0c9fa7a2a2c4ab1b40ea08986a431f486496a7105ecc49aff964bc63746e42fb: Status 404 returned error can't find the container with id 0c9fa7a2a2c4ab1b40ea08986a431f486496a7105ecc49aff964bc63746e42fb
Sep 09 20:12:07.509905 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.509829    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpld5"]
Sep 09 20:12:07.514162 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:07.514130    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8544c883_e15c_40ee_90db_19caf5a8b6ea.slice/crio-7b5ea6ab041b85646b8cb86c26812c80df0628902a7608e86fb89b68830e94b8 WatchSource:0}: Error finding container 7b5ea6ab041b85646b8cb86c26812c80df0628902a7608e86fb89b68830e94b8: Status 404 returned error can't find the container with id 7b5ea6ab041b85646b8cb86c26812c80df0628902a7608e86fb89b68830e94b8
Sep 09 20:12:07.573406 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.572338    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-87c784c5b-k6grx"]
Sep 09 20:12:07.577149 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:07.577084    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f74bf6b_25f7_439d_98c7_1b79b2e89c82.slice/crio-bba0d0018a0422550cb4476e4404c181d065d61f3fee0bdff27e12af67faa0ad WatchSource:0}: Error finding container bba0d0018a0422550cb4476e4404c181d065d61f3fee0bdff27e12af67faa0ad: Status 404 returned error can't find the container with id bba0d0018a0422550cb4476e4404c181d065d61f3fee0bdff27e12af67faa0ad
Sep 09 20:12:07.593183 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.593131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-s6c5t"]
Sep 09 20:12:07.596974 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:07.596946    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8161c5_4de2_42be_b02e_c73ac0a5c015.slice/crio-ba6b241bf1d64370bd69214e72df31a45306e70545660e3d6f0bc04a94904c49 WatchSource:0}: Error finding container ba6b241bf1d64370bd69214e72df31a45306e70545660e3d6f0bc04a94904c49: Status 404 returned error can't find the container with id ba6b241bf1d64370bd69214e72df31a45306e70545660e3d6f0bc04a94904c49
Sep 09 20:12:07.660256 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.660213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55" event={"ID":"d70fa5fc-4c59-48d9-895c-591e860629f3","Type":"ContainerStarted","Data":"1301107cdefbc0130f862feb64d69f0eff952cac5c095cf5eb8b737376ea2c53"}
Sep 09 20:12:07.663093 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.662151    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59679977db-fltqh" event={"ID":"bfe26766-68fa-4503-b62e-bcf304742a08","Type":"ContainerStarted","Data":"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"}
Sep 09 20:12:07.663093 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.662203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59679977db-fltqh" event={"ID":"bfe26766-68fa-4503-b62e-bcf304742a08","Type":"ContainerStarted","Data":"499c512a15f331efff780940e005944690ce3a9c6aee86c1fbea872b564c12f0"}
Sep 09 20:12:07.663093 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.663007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:07.665418 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.665339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t" event={"ID":"fb8161c5-4de2-42be-b02e-c73ac0a5c015","Type":"ContainerStarted","Data":"ba6b241bf1d64370bd69214e72df31a45306e70545660e3d6f0bc04a94904c49"}
Sep 09 20:12:07.668172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.668117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-87c784c5b-k6grx" event={"ID":"5f74bf6b-25f7-439d-98c7-1b79b2e89c82","Type":"ContainerStarted","Data":"802b1a25b5048f49b9b483e5544d517e9bf5d1ca79b111af60296a46dd4d3dd7"}
Sep 09 20:12:07.668172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.668165    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-87c784c5b-k6grx" event={"ID":"5f74bf6b-25f7-439d-98c7-1b79b2e89c82","Type":"ContainerStarted","Data":"bba0d0018a0422550cb4476e4404c181d065d61f3fee0bdff27e12af67faa0ad"}
Sep 09 20:12:07.669681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.669657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpld5" event={"ID":"8544c883-e15c-40ee-90db-19caf5a8b6ea","Type":"ContainerStarted","Data":"7b5ea6ab041b85646b8cb86c26812c80df0628902a7608e86fb89b68830e94b8"}
Sep 09 20:12:07.671397 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.671371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr" event={"ID":"974ebb76-2a57-4c95-971c-6f6be973a3d9","Type":"ContainerStarted","Data":"0c9fa7a2a2c4ab1b40ea08986a431f486496a7105ecc49aff964bc63746e42fb"}
Sep 09 20:12:07.685566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.685503    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59679977db-fltqh" podStartSLOduration=65.685482055 podStartE2EDuration="1m5.685482055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:07.684611421 +0000 UTC m=+67.059149516" watchObservedRunningTime="2026-09-09 20:12:07.685482055 +0000 UTC m=+67.060020151"
Sep 09 20:12:07.707529 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:07.707432    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-87c784c5b-k6grx" podStartSLOduration=38.707408753 podStartE2EDuration="38.707408753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:07.706263569 +0000 UTC m=+67.080801662" watchObservedRunningTime="2026-09-09 20:12:07.707408753 +0000 UTC m=+67.081946848"
Sep 09 20:12:08.289748 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.289720    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:08.293079 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.292851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:08.682104 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.681993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l82g9" event={"ID":"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df","Type":"ContainerStarted","Data":"59e51f6c36c9290e4c3e7837b6fd32cb0226bb53e9cdbd80250ecec2117b03d9"}
Sep 09 20:12:08.682104 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.682088    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:08.682598 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.682120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l82g9" event={"ID":"e2feec63-e0b9-4f53-ac1c-b4e4c876c9df","Type":"ContainerStarted","Data":"ef06fb776d2a88349778fdc5c7ff5897f923f416df1b2669669816145481a3ed"}
Sep 09 20:12:08.684711 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.684641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-87c784c5b-k6grx"
Sep 09 20:12:08.727012 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:08.726569    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l82g9" podStartSLOduration=66.099313024 podStartE2EDuration="1m7.726542799s" totalImagesPullingTime="1.627229775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:08.699914862 +0000 UTC m=+68.074452957" watchObservedRunningTime="2026-09-09 20:12:08.726542799 +0000 UTC m=+68.101080893"
Sep 09 20:12:13.702093 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.702042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55" event={"ID":"d70fa5fc-4c59-48d9-895c-591e860629f3","Type":"ContainerStarted","Data":"eaa263996e80af0309ea0af536f648d59e19c2935327e6254c294fa2b563f2c5"}
Sep 09 20:12:13.702608 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.702120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55" event={"ID":"d70fa5fc-4c59-48d9-895c-591e860629f3","Type":"ContainerStarted","Data":"660e7105f61110c9c14287ec0c0040f73929595d4ddb0e75a382068bc97c158d"}
Sep 09 20:12:13.704269 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.704238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t" event={"ID":"fb8161c5-4de2-42be-b02e-c73ac0a5c015","Type":"ContainerStarted","Data":"22cc5556e7b044a8f417ec917e7370be124ecab9bb4d6ae9f289a25b909ac8ca"}
Sep 09 20:12:13.706192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.706159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rn6b9" event={"ID":"c0a9382c-c5ae-4a35-8e07-09ea6026a485","Type":"ContainerStarted","Data":"3d6ff17b7de45085866e9e3bf4019142d81ba912c7ee436b750871997235e8b8"}
Sep 09 20:12:13.708831 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.708257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpld5" event={"ID":"8544c883-e15c-40ee-90db-19caf5a8b6ea","Type":"ContainerStarted","Data":"9a42ce792ae2561db8bbf17d429110c3a824e18754fc27c2ee781bdcf741a11f"}
Sep 09 20:12:13.708831 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.708309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpld5" event={"ID":"8544c883-e15c-40ee-90db-19caf5a8b6ea","Type":"ContainerStarted","Data":"e63195f5ba28af837ccc9b63a65a70e801160856553c84422fca3b42e438262b"}
Sep 09 20:12:13.708831 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.708775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kpld5"
Sep 09 20:12:13.710980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.710547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr" event={"ID":"974ebb76-2a57-4c95-971c-6f6be973a3d9","Type":"ContainerStarted","Data":"b717f9538a943e94c84817c0ac0f8ce82301f6073284b8ad1adfb5ae56b024b4"}
Sep 09 20:12:13.721913 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.721846    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mgs55" podStartSLOduration=38.898751036 podStartE2EDuration="44.721822812s" totalImagesPullingTime="5.823071776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:13.720180284 +0000 UTC m=+73.094718378" watchObservedRunningTime="2026-09-09 20:12:13.721822812 +0000 UTC m=+73.096360906"
Sep 09 20:12:13.739310 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.739248    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rn6b9" podStartSLOduration=64.908012009 podStartE2EDuration="1m11.739228432s" totalImagesPullingTime="6.831216423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:13.738987355 +0000 UTC m=+73.113525451" watchObservedRunningTime="2026-09-09 20:12:13.739228432 +0000 UTC m=+73.113766526"
Sep 09 20:12:13.756195 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.756125    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kpld5" podStartSLOduration=34.004152978 podStartE2EDuration="39.756106527s" totalImagesPullingTime="5.751953549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:13.755278666 +0000 UTC m=+73.129816761" watchObservedRunningTime="2026-09-09 20:12:13.756106527 +0000 UTC m=+73.130644621"
Sep 09 20:12:13.774551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.774497    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-s6c5t" podStartSLOduration=33.292446898 podStartE2EDuration="38.774480197s" totalImagesPullingTime="5.482033299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:13.77423266 +0000 UTC m=+73.148770757" watchObservedRunningTime="2026-09-09 20:12:13.774480197 +0000 UTC m=+73.149018286"
Sep 09 20:12:13.791172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:13.791028    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9zrpr" podStartSLOduration=39.012105157 podStartE2EDuration="44.791007005s" totalImagesPullingTime="5.778901848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:13.790860501 +0000 UTC m=+73.165398597" watchObservedRunningTime="2026-09-09 20:12:13.791007005 +0000 UTC m=+73.165545101"
Sep 09 20:12:18.113740 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.113706    2564 scope.go:117] "RemoveContainer" containerID="c93a23d2fb3680937e739e8d7d0cebb14201fde639c109fe5dc07cbf2c766702"
Sep 09 20:12:18.730515 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.730484    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:12:18.730696 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.730590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" event={"ID":"cdea4a8f-b525-4646-b6dd-39da4ba10271","Type":"ContainerStarted","Data":"b809205699c3feb347f6c704d631236276a7f4a783e55cf2c612ef8039998ac0"}
Sep 09 20:12:18.730944 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.730922    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:12:18.735924 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.735905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-lmvxs"
Sep 09 20:12:18.752412 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:18.752360    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-lmvxs" podStartSLOduration=42.103953421 podStartE2EDuration="49.752343305s" totalImagesPullingTime="7.648389884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:18.751472758 +0000 UTC m=+78.126010853" watchObservedRunningTime="2026-09-09 20:12:18.752343305 +0000 UTC m=+78.126881396"
Sep 09 20:12:23.334907 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.334779    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"]
Sep 09 20:12:23.337507 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.337489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.342834 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.342802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:12:23.342974 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.342813    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:23.342974 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.342855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wpg2n\""
Sep 09 20:12:23.343624 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.343611    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:23.348719 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.348682    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"]
Sep 09 20:12:23.351467 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.351447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"]
Sep 09 20:12:23.351585 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.351538    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.353978 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.353954    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:23.354150 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.354106    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 20:12:23.354150 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.353957    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 20:12:23.354359 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.354343    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hg7vr\""
Sep 09 20:12:23.365324 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.365302    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"]
Sep 09 20:12:23.375029 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.375007    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mj7fv"]
Sep 09 20:12:23.377540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.377524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.379642 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.379613    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:23.379757 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.379732    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zl8h2\""
Sep 09 20:12:23.379968 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.379949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:23.380879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.380860    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:23.498592 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.498592 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-wtmp\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498637    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-root\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9z7l\" (UniqueName: \"kubernetes.io/projected/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-api-access-l9z7l\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.498801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498888    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f6ad1d-cdc5-424f-9685-880228aa6251-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-textfile\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbpk\" (UniqueName: \"kubernetes.io/projected/c58100be-3a65-4ad2-86ee-78288a119029-kube-api-access-dsbpk\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.498974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.499031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-metrics-client-ca\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.499088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppmd\" (UniqueName: \"kubernetes.io/projected/53f6ad1d-cdc5-424f-9685-880228aa6251-kube-api-access-lppmd\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.499134 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.499122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-sys\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.499465 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.499150    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.600559 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.600559 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.600559 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.600838 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.600838 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f6ad1d-cdc5-424f-9685-880228aa6251-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.600939 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-textfile\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.600939 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.600939 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbpk\" (UniqueName: \"kubernetes.io/projected/c58100be-3a65-4ad2-86ee-78288a119029-kube-api-access-dsbpk\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.600954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-metrics-client-ca\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppmd\" (UniqueName: \"kubernetes.io/projected/53f6ad1d-cdc5-424f-9685-880228aa6251-kube-api-access-lppmd\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.601126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-sys\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601185    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-wtmp\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-root\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601363 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9z7l\" (UniqueName: \"kubernetes.io/projected/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-api-access-l9z7l\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601389    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601479    2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53f6ad1d-cdc5-424f-9685-880228aa6251-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601583    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls podName:53f6ad1d-cdc5-424f-9685-880228aa6251 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:24.10155452 +0000 UTC m=+83.476092611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-jhrzr" (UID: "53f6ad1d-cdc5-424f-9685-880228aa6251") : secret "openshift-state-metrics-tls" not found
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-wtmp\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.601686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-root\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601722    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601784    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls podName:44aebd4a-7a3b-41b6-899f-ac5655cf7ec7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:24.101766956 +0000 UTC m=+83.476305044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-lbtd4" (UID: "44aebd4a-7a3b-41b6-899f-ac5655cf7ec7") : secret "kube-state-metrics-tls" not found
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601813    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c58100be-3a65-4ad2-86ee-78288a119029-sys\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:23.601878    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls podName:c58100be-3a65-4ad2-86ee-78288a119029 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:24.101857944 +0000 UTC m=+83.476396034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls") pod "node-exporter-mj7fv" (UID: "c58100be-3a65-4ad2-86ee-78288a119029") : secret "node-exporter-tls" not found
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58100be-3a65-4ad2-86ee-78288a119029-metrics-client-ca\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.602102 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.601999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-textfile\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.602464 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.602249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.603555 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.603526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.604218 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.604199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:23.604218 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.604203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.612627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.612600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbpk\" (UniqueName: \"kubernetes.io/projected/c58100be-3a65-4ad2-86ee-78288a119029-kube-api-access-dsbpk\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:23.613461 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.613436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9z7l\" (UniqueName: \"kubernetes.io/projected/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-api-access-l9z7l\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:23.613569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:23.613436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppmd\" (UniqueName: \"kubernetes.io/projected/53f6ad1d-cdc5-424f-9685-880228aa6251-kube-api-access-lppmd\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:24.106772 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.106729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:24.106970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.106795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:24.106970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.106865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:24.109142 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.109112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c58100be-3a65-4ad2-86ee-78288a119029-node-exporter-tls\") pod \"node-exporter-mj7fv\" (UID: \"c58100be-3a65-4ad2-86ee-78288a119029\") " pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:24.109255 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.109148    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53f6ad1d-cdc5-424f-9685-880228aa6251-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-jhrzr\" (UID: \"53f6ad1d-cdc5-424f-9685-880228aa6251\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:24.109327 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.109318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44aebd4a-7a3b-41b6-899f-ac5655cf7ec7-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-lbtd4\" (UID: \"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:24.251205 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.251168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"
Sep 09 20:12:24.260229 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.260110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"
Sep 09 20:12:24.287916 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.287791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mj7fv"
Sep 09 20:12:24.300851 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:24.300817    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58100be_3a65_4ad2_86ee_78288a119029.slice/crio-cc19b678ef4ab7bbab4c63bc7c383a252f2153c5e037f048023b990e967797a1 WatchSource:0}: Error finding container cc19b678ef4ab7bbab4c63bc7c383a252f2153c5e037f048023b990e967797a1: Status 404 returned error can't find the container with id cc19b678ef4ab7bbab4c63bc7c383a252f2153c5e037f048023b990e967797a1
Sep 09 20:12:24.536640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.536608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr"]
Sep 09 20:12:24.539543 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:24.539498    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f6ad1d_cdc5_424f_9685_880228aa6251.slice/crio-b4f8d506e08c28146da71c25170bb279888d0b88d03762883fac3df0a46166b9 WatchSource:0}: Error finding container b4f8d506e08c28146da71c25170bb279888d0b88d03762883fac3df0a46166b9: Status 404 returned error can't find the container with id b4f8d506e08c28146da71c25170bb279888d0b88d03762883fac3df0a46166b9
Sep 09 20:12:24.541678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.541646    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4"]
Sep 09 20:12:24.547117 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:24.547087    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aebd4a_7a3b_41b6_899f_ac5655cf7ec7.slice/crio-eedfc1fe16a7fb48d7ae47d761d53334fcb35adcf562599f4652dea0d2559c7b WatchSource:0}: Error finding container eedfc1fe16a7fb48d7ae47d761d53334fcb35adcf562599f4652dea0d2559c7b: Status 404 returned error can't find the container with id eedfc1fe16a7fb48d7ae47d761d53334fcb35adcf562599f4652dea0d2559c7b
Sep 09 20:12:24.721174 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.721144    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kpld5"
Sep 09 20:12:24.757940 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.757835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4" event={"ID":"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7","Type":"ContainerStarted","Data":"eedfc1fe16a7fb48d7ae47d761d53334fcb35adcf562599f4652dea0d2559c7b"}
Sep 09 20:12:24.759970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.759875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj7fv" event={"ID":"c58100be-3a65-4ad2-86ee-78288a119029","Type":"ContainerStarted","Data":"cc19b678ef4ab7bbab4c63bc7c383a252f2153c5e037f048023b990e967797a1"}
Sep 09 20:12:24.762511 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.762474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr" event={"ID":"53f6ad1d-cdc5-424f-9685-880228aa6251","Type":"ContainerStarted","Data":"facbcc79257c5757be1c069a339e3282d4dd4afce6cd70b8ed4b413486be7bf4"}
Sep 09 20:12:24.762666 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.762547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr" event={"ID":"53f6ad1d-cdc5-424f-9685-880228aa6251","Type":"ContainerStarted","Data":"1e0759aa7b6810df2f08ff6709a1b5831cd39a2a218d39130dc32bf6f1ee566c"}
Sep 09 20:12:24.762666 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:24.762577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr" event={"ID":"53f6ad1d-cdc5-424f-9685-880228aa6251","Type":"ContainerStarted","Data":"b4f8d506e08c28146da71c25170bb279888d0b88d03762883fac3df0a46166b9"}
Sep 09 20:12:25.770931 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:25.770853    2564 generic.go:356] "Generic (PLEG): container finished" podID="c58100be-3a65-4ad2-86ee-78288a119029" containerID="76c2ad92eb341592024a19103a560e039aa37a9328002cc285c7d030b8192dd9" exitCode=0
Sep 09 20:12:25.771426 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:25.770941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj7fv" event={"ID":"c58100be-3a65-4ad2-86ee-78288a119029","Type":"ContainerDied","Data":"76c2ad92eb341592024a19103a560e039aa37a9328002cc285c7d030b8192dd9"}
Sep 09 20:12:26.778081 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.777963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj7fv" event={"ID":"c58100be-3a65-4ad2-86ee-78288a119029","Type":"ContainerStarted","Data":"3a296955ba08876a35b84210bd3cb584ea674eea8b8a83d4e746aff7bc9bd789"}
Sep 09 20:12:26.778081 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.778037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj7fv" event={"ID":"c58100be-3a65-4ad2-86ee-78288a119029","Type":"ContainerStarted","Data":"243fa17dbc15c373e328be338578f594d7aea5a76be55dcdc0cbc35d89e800b9"}
Sep 09 20:12:26.780450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.780415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr" event={"ID":"53f6ad1d-cdc5-424f-9685-880228aa6251","Type":"ContainerStarted","Data":"a864dc1eac77e22270c8b9e2229effb37efed11c819258f8d4dda042bd5ba13e"}
Sep 09 20:12:26.782771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.782736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4" event={"ID":"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7","Type":"ContainerStarted","Data":"eb040ed0728409a2e5ebaf04ee2eba79c6ce716a2eff2fdfbcf9f37a91d59f7b"}
Sep 09 20:12:26.782891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.782790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4" event={"ID":"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7","Type":"ContainerStarted","Data":"09d15a4cdf3ca69d8486570d9110ef8088ab8bbfe373a9346e47fbfee242fa39"}
Sep 09 20:12:26.782891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.782827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4" event={"ID":"44aebd4a-7a3b-41b6-899f-ac5655cf7ec7","Type":"ContainerStarted","Data":"589e4666146632785fa77819e4149700a18b174906793f430364b2d264d97bb9"}
Sep 09 20:12:26.798307 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.798254    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mj7fv" podStartSLOduration=2.909107117 podStartE2EDuration="3.798236602s" totalImagesPullingTime="819.864519ms" totalInitContainerRuntime="69.264966ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.797228977 +0000 UTC m=+86.171767073" watchObservedRunningTime="2026-09-09 20:12:26.798236602 +0000 UTC m=+86.172774696"
Sep 09 20:12:26.814496 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.814444    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-jhrzr" podStartSLOduration=2.13042198 podStartE2EDuration="3.814431331s" totalImagesPullingTime="1.684009351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.812781592 +0000 UTC m=+86.187319688" watchObservedRunningTime="2026-09-09 20:12:26.814431331 +0000 UTC m=+86.188969425"
Sep 09 20:12:26.830858 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:26.830798    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-lbtd4" podStartSLOduration=2.035129074 podStartE2EDuration="3.830779971s" totalImagesPullingTime="1.795650897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.829377954 +0000 UTC m=+86.203916049" watchObservedRunningTime="2026-09-09 20:12:26.830779971 +0000 UTC m=+86.205318065"
Sep 09 20:12:28.097764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.097727    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"]
Sep 09 20:12:28.100028 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.100007    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.102240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.102216    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 20:12:28.102314 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.102262    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nz6pf\""
Sep 09 20:12:28.111879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.111847    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"]
Sep 09 20:12:28.243493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.243457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.243493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.243494    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zckh\" (UniqueName: \"kubernetes.io/projected/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-kube-api-access-5zckh\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.343915 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.343863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.343915 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.343905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zckh\" (UniqueName: \"kubernetes.io/projected/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-kube-api-access-5zckh\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.344152 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:28.344014    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 20:12:28.344152 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:28.344105    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert podName:cbf01d2a-e87c-467a-8d0e-c60ce9736cec nodeName:}" failed. No retries permitted until 2026-09-09 20:12:28.844087976 +0000 UTC m=+88.218626049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-4rjzm" (UID: "cbf01d2a-e87c-467a-8d0e-c60ce9736cec") : secret "monitoring-plugin-cert" not found
Sep 09 20:12:28.354922 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.354847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zckh\" (UniqueName: \"kubernetes.io/projected/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-kube-api-access-5zckh\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.547467 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.547433    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75db8b8f67-m2drk"]
Sep 09 20:12:28.550131 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.550103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.552415 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552377    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:12:28.552566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552454    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:12:28.552566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552463    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:12:28.552566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552540    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:12:28.552566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8s9ph\""
Sep 09 20:12:28.552761 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.552672    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:12:28.556688 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.556665    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:12:28.564447 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.564421    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75db8b8f67-m2drk"]
Sep 09 20:12:28.646166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-metrics-client-ca\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-serving-certs-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-federate-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.646384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.646348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkwj\" (UniqueName: \"kubernetes.io/projected/6ad2ea8f-ea87-444a-9781-6cee9319ab07-kube-api-access-5tkwj\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkwj\" (UniqueName: \"kubernetes.io/projected/6ad2ea8f-ea87-444a-9781-6cee9319ab07-kube-api-access-5tkwj\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-metrics-client-ca\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747651 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-serving-certs-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747848 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747910 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-federate-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.747910 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.747889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.748292 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.748263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-metrics-client-ca\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.748428 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.748399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-serving-certs-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.748522 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.748499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.750269 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.750247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.750466 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.750443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-federate-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.750533 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.750490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-secret-telemeter-client\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.750533 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.750509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ad2ea8f-ea87-444a-9781-6cee9319ab07-telemeter-client-tls\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.755514 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.755492    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkwj\" (UniqueName: \"kubernetes.io/projected/6ad2ea8f-ea87-444a-9781-6cee9319ab07-kube-api-access-5tkwj\") pod \"telemeter-client-75db8b8f67-m2drk\" (UID: \"6ad2ea8f-ea87-444a-9781-6cee9319ab07\") " pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:28.848872 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.848818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.851266 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.851236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbf01d2a-e87c-467a-8d0e-c60ce9736cec-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-4rjzm\" (UID: \"cbf01d2a-e87c-467a-8d0e-c60ce9736cec\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:28.860822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:28.860799    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk"
Sep 09 20:12:29.009809 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.009777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:29.138692 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.138665    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75db8b8f67-m2drk"]
Sep 09 20:12:29.140682 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:29.140659    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad2ea8f_ea87_444a_9781_6cee9319ab07.slice/crio-c0b269f2e8758f3bf5c2e73af7f5a4441c7ea56153960b995bb454201746ab6f WatchSource:0}: Error finding container c0b269f2e8758f3bf5c2e73af7f5a4441c7ea56153960b995bb454201746ab6f: Status 404 returned error can't find the container with id c0b269f2e8758f3bf5c2e73af7f5a4441c7ea56153960b995bb454201746ab6f
Sep 09 20:12:29.279948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.279788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"]
Sep 09 20:12:29.282708 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:29.282663    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf01d2a_e87c_467a_8d0e_c60ce9736cec.slice/crio-ab1e7cf957649aa2cbb4cfde5bc8a047d0fb0942e8b3fd3f84af8d5233fafa8c WatchSource:0}: Error finding container ab1e7cf957649aa2cbb4cfde5bc8a047d0fb0942e8b3fd3f84af8d5233fafa8c: Status 404 returned error can't find the container with id ab1e7cf957649aa2cbb4cfde5bc8a047d0fb0942e8b3fd3f84af8d5233fafa8c
Sep 09 20:12:29.690052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.689825    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:29.800997 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.800679    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:12:29.803327 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.803292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm" event={"ID":"cbf01d2a-e87c-467a-8d0e-c60ce9736cec","Type":"ContainerStarted","Data":"ab1e7cf957649aa2cbb4cfde5bc8a047d0fb0942e8b3fd3f84af8d5233fafa8c"}
Sep 09 20:12:29.803470 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.803400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.805166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.803738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk" event={"ID":"6ad2ea8f-ea87-444a-9781-6cee9319ab07","Type":"ContainerStarted","Data":"c0b269f2e8758f3bf5c2e73af7f5a4441c7ea56153960b995bb454201746ab6f"}
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.805652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.805777    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806020    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806324    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh2nm\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806530    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806020    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:12:29.806724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806605    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:12:29.807159 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.806833    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:12:29.810253 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.810228    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:12:29.960508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.960508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.960726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4752f\" (UniqueName: \"kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.960726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.960726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:29.960726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:29.960669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.061919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4752f\" (UniqueName: \"kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.061987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.062078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.062169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.062195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.062222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.062970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.062945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.063147 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.063054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.063147 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.063102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.065392 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.065368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.065654 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.065637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.070635 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.070609    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4752f\" (UniqueName: \"kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f\") pod \"console-5848777cfd-pdp7m\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") " pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.117766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.117730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:30.402969 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.402943    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:12:30.516399 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:12:30.516366    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1403e7_0f4f_4e3d_8d1a_18a0d987d948.slice/crio-1472b376d4ef11519734f459de13a87acd6b857139b0b26c3be0f9ea95353205 WatchSource:0}: Error finding container 1472b376d4ef11519734f459de13a87acd6b857139b0b26c3be0f9ea95353205: Status 404 returned error can't find the container with id 1472b376d4ef11519734f459de13a87acd6b857139b0b26c3be0f9ea95353205
Sep 09 20:12:30.809594 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:30.809553    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5848777cfd-pdp7m" event={"ID":"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948","Type":"ContainerStarted","Data":"1472b376d4ef11519734f459de13a87acd6b857139b0b26c3be0f9ea95353205"}
Sep 09 20:12:31.390773 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.390698    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:12:31.817736 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.817693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm" event={"ID":"cbf01d2a-e87c-467a-8d0e-c60ce9736cec","Type":"ContainerStarted","Data":"6685ebb3eceed64174896b3fee6d14467ae33f252d46704d08626916d9b66b37"}
Sep 09 20:12:31.818187 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.817967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:31.821130 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.821096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk" event={"ID":"6ad2ea8f-ea87-444a-9781-6cee9319ab07","Type":"ContainerStarted","Data":"e9e90157b11a8e2c5826eb0200c3c1c4aa303a5d4f8ff351476640b075cf6b7c"}
Sep 09 20:12:31.825001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.824973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm"
Sep 09 20:12:31.837382 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:31.837331    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-4rjzm" podStartSLOduration=2.138429197 podStartE2EDuration="3.837309683s" totalImagesPullingTime="1.698880486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:31.835272168 +0000 UTC m=+91.209810263" watchObservedRunningTime="2026-09-09 20:12:31.837309683 +0000 UTC m=+91.211847780"
Sep 09 20:12:32.828365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:32.828016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk" event={"ID":"6ad2ea8f-ea87-444a-9781-6cee9319ab07","Type":"ContainerStarted","Data":"6871ae1638a6ba0129ab6af4ec5660c31f025d33edc211201c375932a93aa626"}
Sep 09 20:12:32.828365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:32.828097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk" event={"ID":"6ad2ea8f-ea87-444a-9781-6cee9319ab07","Type":"ContainerStarted","Data":"d75276b3deb63962080bbb53a6e0922b9f1171265cd7420e1bd05d8387c848e6"}
Sep 09 20:12:32.849598 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:32.849527    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75db8b8f67-m2drk" podStartSLOduration=1.857392708 podStartE2EDuration="4.849499786s" totalImagesPullingTime="2.992107078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:32.847805681 +0000 UTC m=+92.222343776" watchObservedRunningTime="2026-09-09 20:12:32.849499786 +0000 UTC m=+92.224037874"
Sep 09 20:12:33.834129 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:33.834032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5848777cfd-pdp7m" event={"ID":"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948","Type":"ContainerStarted","Data":"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"}
Sep 09 20:12:33.851203 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:33.851132    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5848777cfd-pdp7m" podStartSLOduration=1.8275400469999998 podStartE2EDuration="4.851115284s" totalImagesPullingTime="3.023575237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:33.849458216 +0000 UTC m=+93.223996311" watchObservedRunningTime="2026-09-09 20:12:33.851115284 +0000 UTC m=+93.225653378"
Sep 09 20:12:37.674158 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:37.674125    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mtz8b"
Sep 09 20:12:40.118838 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:40.118792    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:40.119262 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:40.118866    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:40.123809 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:40.123784    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:40.865345 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:40.865315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:12:50.689380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:50.689336    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:12:56.420986 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.420916    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59679977db-fltqh" podUID="bfe26766-68fa-4503-b62e-bcf304742a08" containerName="registry" containerID="cri-o://66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6" gracePeriod=30
Sep 09 20:12:56.659098 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.659050    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:56.791586 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791550    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791761 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791606    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791761 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791651    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwrzk\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791880 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791795    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791880 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791974 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.791974 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.791930    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.792095 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.792031    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets\") pod \"bfe26766-68fa-4503-b62e-bcf304742a08\" (UID: \"bfe26766-68fa-4503-b62e-bcf304742a08\") "
Sep 09 20:12:56.792512 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.792427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:56.792713 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.792684    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:56.794481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.794452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:56.794599 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.794559    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:56.794657 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.794594    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:56.794730 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.794713    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:56.794774 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.794725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk" (OuterVolumeSpecName: "kube-api-access-mwrzk") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "kube-api-access-mwrzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:56.800378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.800339    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bfe26766-68fa-4503-b62e-bcf304742a08" (UID: "bfe26766-68fa-4503-b62e-bcf304742a08"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:56.893265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893230    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-installation-pull-secrets\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893260    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfe26766-68fa-4503-b62e-bcf304742a08-image-registry-private-configuration\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893273    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-registry-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893282    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrzk\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-kube-api-access-mwrzk\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893291    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfe26766-68fa-4503-b62e-bcf304742a08-bound-sa-token\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893300    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfe26766-68fa-4503-b62e-bcf304742a08-ca-trust-extracted\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893308    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-trusted-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.893481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.893316    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfe26766-68fa-4503-b62e-bcf304742a08-registry-certificates\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:12:56.926900 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.926867    2564 generic.go:356] "Generic (PLEG): container finished" podID="bfe26766-68fa-4503-b62e-bcf304742a08" containerID="66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6" exitCode=0
Sep 09 20:12:56.927126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.926910    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59679977db-fltqh"
Sep 09 20:12:56.927126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.926953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59679977db-fltqh" event={"ID":"bfe26766-68fa-4503-b62e-bcf304742a08","Type":"ContainerDied","Data":"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"}
Sep 09 20:12:56.927126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.927042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59679977db-fltqh" event={"ID":"bfe26766-68fa-4503-b62e-bcf304742a08","Type":"ContainerDied","Data":"499c512a15f331efff780940e005944690ce3a9c6aee86c1fbea872b564c12f0"}
Sep 09 20:12:56.927126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.927087    2564 scope.go:117] "RemoveContainer" containerID="66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"
Sep 09 20:12:56.936612 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.936592    2564 scope.go:117] "RemoveContainer" containerID="66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"
Sep 09 20:12:56.936894 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:12:56.936876    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6\": container with ID starting with 66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6 not found: ID does not exist" containerID="66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"
Sep 09 20:12:56.936932 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.936903    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6"} err="failed to get container status \"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6\": rpc error: code = NotFound desc = could not find container \"66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6\": container with ID starting with 66011b3213113e12b02408414630f4ff52856227515d68680351a8beef6813c6 not found: ID does not exist"
Sep 09 20:12:56.947471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.947440    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:12:56.951695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:56.951674    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59679977db-fltqh"]
Sep 09 20:12:57.118101 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:57.118002    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe26766-68fa-4503-b62e-bcf304742a08" path="/var/lib/kubelet/pods/bfe26766-68fa-4503-b62e-bcf304742a08/volumes"
Sep 09 20:12:59.941537 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:59.941503    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e5dfd22-9ef7-488b-9819-5eb4fbba4035" containerID="84ad69dc0c97dcbf0ff69494de4bc52a837b3911f4204cf49ad93a12cbb62db2" exitCode=0
Sep 09 20:12:59.941932 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:59.941571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g" event={"ID":"8e5dfd22-9ef7-488b-9819-5eb4fbba4035","Type":"ContainerDied","Data":"84ad69dc0c97dcbf0ff69494de4bc52a837b3911f4204cf49ad93a12cbb62db2"}
Sep 09 20:12:59.941979 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:12:59.941949    2564 scope.go:117] "RemoveContainer" containerID="84ad69dc0c97dcbf0ff69494de4bc52a837b3911f4204cf49ad93a12cbb62db2"
Sep 09 20:13:00.947667 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:00.947634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dcg2g" event={"ID":"8e5dfd22-9ef7-488b-9819-5eb4fbba4035","Type":"ContainerStarted","Data":"7408c898ccfeb37615a3a79d48ecf4541963e925a2b1195cb12f312b54a43b98"}
Sep 09 20:13:15.005027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:15.004937    2564 generic.go:356] "Generic (PLEG): container finished" podID="aba3eb39-9f90-4766-be49-f459acbf5379" containerID="8aa79faa851e0cbf74d6da373f7bbb4b7ddb7281dfc5a325fd32ca3e6ec9c9d2" exitCode=0
Sep 09 20:13:15.005027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:15.005009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8" event={"ID":"aba3eb39-9f90-4766-be49-f459acbf5379","Type":"ContainerDied","Data":"8aa79faa851e0cbf74d6da373f7bbb4b7ddb7281dfc5a325fd32ca3e6ec9c9d2"}
Sep 09 20:13:15.005445 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:15.005411    2564 scope.go:117] "RemoveContainer" containerID="8aa79faa851e0cbf74d6da373f7bbb4b7ddb7281dfc5a325fd32ca3e6ec9c9d2"
Sep 09 20:13:15.710799 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:15.710724    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5848777cfd-pdp7m" podUID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" containerName="console" containerID="cri-o://f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3" gracePeriod=15
Sep 09 20:13:15.954587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:15.954562    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:13:16.012476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.012383    2564 generic.go:356] "Generic (PLEG): container finished" podID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" containerID="f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3" exitCode=0
Sep 09 20:13:16.012476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.012435    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5848777cfd-pdp7m"
Sep 09 20:13:16.012476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.012460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5848777cfd-pdp7m" event={"ID":"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948","Type":"ContainerDied","Data":"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"}
Sep 09 20:13:16.013154 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.012510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5848777cfd-pdp7m" event={"ID":"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948","Type":"ContainerDied","Data":"1472b376d4ef11519734f459de13a87acd6b857139b0b26c3be0f9ea95353205"}
Sep 09 20:13:16.013154 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.012534    2564 scope.go:117] "RemoveContainer" containerID="f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"
Sep 09 20:13:16.015939 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.015363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nh9z8" event={"ID":"aba3eb39-9f90-4766-be49-f459acbf5379","Type":"ContainerStarted","Data":"fd2fdda464fafdb2c4abb74bf5ed00420ee9d8ead60bc681c9070905a202a162"}
Sep 09 20:13:16.023039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.023014    2564 scope.go:117] "RemoveContainer" containerID="f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"
Sep 09 20:13:16.023355 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:13:16.023334    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3\": container with ID starting with f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3 not found: ID does not exist" containerID="f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"
Sep 09 20:13:16.023427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.023363    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3"} err="failed to get container status \"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3\": rpc error: code = NotFound desc = could not find container \"f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3\": container with ID starting with f6270cc63c39edb4dc441f3f63325ddbeecd7c45b0340401af9b7db95b0dadb3 not found: ID does not exist"
Sep 09 20:13:16.062739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062706    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.062927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062752    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4752f\" (UniqueName: \"kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.062927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.062927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062839    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.062927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062862    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.062927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.062908    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config\") pod \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\" (UID: \"7d1403e7-0f4f-4e3d-8d1a-18a0d987d948\") "
Sep 09 20:13:16.063350 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.063319    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:16.063350 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.063332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:16.063540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.063400    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config" (OuterVolumeSpecName: "console-config") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:16.065678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.065647    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f" (OuterVolumeSpecName: "kube-api-access-4752f") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "kube-api-access-4752f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:16.065678 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.065647    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:16.065819 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.065740    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" (UID: "7d1403e7-0f4f-4e3d-8d1a-18a0d987d948"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:16.163864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163823    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.163864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163857    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.163864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163871    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4752f\" (UniqueName: \"kubernetes.io/projected/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-kube-api-access-4752f\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.164151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163885    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.164151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163898    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.164151 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.163910    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:13:16.332731 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.332698    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:13:16.337335 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:16.337303    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5848777cfd-pdp7m"]
Sep 09 20:13:17.118364 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:17.118331    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" path="/var/lib/kubelet/pods/7d1403e7-0f4f-4e3d-8d1a-18a0d987d948/volumes"
Sep 09 20:13:20.034748 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:20.034709    2564 generic.go:356] "Generic (PLEG): container finished" podID="3affe817-470e-426a-aeae-1e75a35602dc" containerID="89fd2a0ed43423f1181e6d94ea15a70289dd5c749f15d31825e0f9e845a509f3" exitCode=0
Sep 09 20:13:20.035141 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:20.034781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz" event={"ID":"3affe817-470e-426a-aeae-1e75a35602dc","Type":"ContainerDied","Data":"89fd2a0ed43423f1181e6d94ea15a70289dd5c749f15d31825e0f9e845a509f3"}
Sep 09 20:13:20.035193 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:20.035161    2564 scope.go:117] "RemoveContainer" containerID="89fd2a0ed43423f1181e6d94ea15a70289dd5c749f15d31825e0f9e845a509f3"
Sep 09 20:13:21.041275 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:21.041241    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-x8nlz" event={"ID":"3affe817-470e-426a-aeae-1e75a35602dc","Type":"ContainerStarted","Data":"28e1517dfe432c8eb507ddf329f72e07f5a43ea655452a352af518fe1be66c59"}
Sep 09 20:13:43.921825 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.921787    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:13:43.922923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.922896    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" containerName="console"
Sep 09 20:13:43.923087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.923074    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" containerName="console"
Sep 09 20:13:43.923189 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.923179    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe26766-68fa-4503-b62e-bcf304742a08" containerName="registry"
Sep 09 20:13:43.923258 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.923249    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe26766-68fa-4503-b62e-bcf304742a08" containerName="registry"
Sep 09 20:13:43.923432 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.923420    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe26766-68fa-4503-b62e-bcf304742a08" containerName="registry"
Sep 09 20:13:43.923532 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.923522    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1403e7-0f4f-4e3d-8d1a-18a0d987d948" containerName="console"
Sep 09 20:13:43.928471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.928440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.930894 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.930870    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:13:43.930894 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.930875    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:13:43.931209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.930935    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:13:43.931209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.930954    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:13:43.931209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.931049    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:13:43.931565 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.931545    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:13:43.931681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.931660    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:13:43.931835 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.931818    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh2nm\""
Sep 09 20:13:43.936265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.936237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:13:43.937741 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.937716    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:13:43.991619 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991646    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991719    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89fd\" (UniqueName: \"kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991771    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:43.991966 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:43.991798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.092917 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.092881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.092929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.092950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89fd\" (UniqueName: \"kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093406 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093802 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093906 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093943 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.093977 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.093947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.095516 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.095492    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.095630 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.095612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.100847 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.100827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89fd\" (UniqueName: \"kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd\") pod \"console-5f5b77db46-jbdzx\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") " pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.240220 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.240126    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:44.511408 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:44.511382    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:13:44.513960 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:13:44.513936    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd906771d_4945_4d55_adcd_041d6cfcc898.slice/crio-97c93933f86e967e006441774a99e2a799d1f9e11585325a311a52150842552a WatchSource:0}: Error finding container 97c93933f86e967e006441774a99e2a799d1f9e11585325a311a52150842552a: Status 404 returned error can't find the container with id 97c93933f86e967e006441774a99e2a799d1f9e11585325a311a52150842552a
Sep 09 20:13:45.147753 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:45.147718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b77db46-jbdzx" event={"ID":"d906771d-4945-4d55-adcd-041d6cfcc898","Type":"ContainerStarted","Data":"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"}
Sep 09 20:13:45.148230 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:45.147769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b77db46-jbdzx" event={"ID":"d906771d-4945-4d55-adcd-041d6cfcc898","Type":"ContainerStarted","Data":"97c93933f86e967e006441774a99e2a799d1f9e11585325a311a52150842552a"}
Sep 09 20:13:45.165688 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:45.165632    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f5b77db46-jbdzx" podStartSLOduration=2.165614086 podStartE2EDuration="2.165614086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:45.164206851 +0000 UTC m=+164.538744946" watchObservedRunningTime="2026-09-09 20:13:45.165614086 +0000 UTC m=+164.540152180"
Sep 09 20:13:54.241217 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:54.241131    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:54.241217 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:54.241189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:54.246020 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:54.245994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:13:55.190109 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:13:55.190078    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:16:00.997695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:00.997667    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:16:00.999969 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:00.999946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:16:01.004685 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:01.004658    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:16:01.006698 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:01.006675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:16:01.008864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:01.008846    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:06.276582 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.276549    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"]
Sep 09 20:16:06.279862 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.279846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.282353 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.282330    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:16:06.282989 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.282974    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:16:06.283084 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.283047    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tnlb4\""
Sep 09 20:16:06.290481 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.290460    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"]
Sep 09 20:16:06.396192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.396147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7n7b\" (UniqueName: \"kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.396380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.396292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.396380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.396335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.497026 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.496984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.497225 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.497033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.497225 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.497105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7n7b\" (UniqueName: \"kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.497465 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.497440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.497504 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.497449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.509527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.509501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7n7b\" (UniqueName: \"kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.589368 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.589333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:06.862908 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.862883    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"]
Sep 09 20:16:06.865414 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:16:06.865387    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd496e0d8_645a_4760_8242_3ca54bbcfd00.slice/crio-56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59 WatchSource:0}: Error finding container 56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59: Status 404 returned error can't find the container with id 56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59
Sep 09 20:16:06.867478 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:06.867455    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:07.716997 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:07.716955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22" event={"ID":"d496e0d8-645a-4760-8242-3ca54bbcfd00","Type":"ContainerStarted","Data":"56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59"}
Sep 09 20:16:13.744024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:13.743985    2564 generic.go:356] "Generic (PLEG): container finished" podID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerID="487e960024cea3e9145aa4e851430dcfa2e28f218bf43ef8cf80aae145d12cd2" exitCode=0
Sep 09 20:16:13.744466 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:13.744100    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22" event={"ID":"d496e0d8-645a-4760-8242-3ca54bbcfd00","Type":"ContainerDied","Data":"487e960024cea3e9145aa4e851430dcfa2e28f218bf43ef8cf80aae145d12cd2"}
Sep 09 20:16:16.759364 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:16.759326    2564 generic.go:356] "Generic (PLEG): container finished" podID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerID="72480de127a8b0757393f32ce53b8bea88762635503562917cf753741feec127" exitCode=0
Sep 09 20:16:16.759750 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:16.759406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22" event={"ID":"d496e0d8-645a-4760-8242-3ca54bbcfd00","Type":"ContainerDied","Data":"72480de127a8b0757393f32ce53b8bea88762635503562917cf753741feec127"}
Sep 09 20:16:24.797274 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:24.797237    2564 generic.go:356] "Generic (PLEG): container finished" podID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerID="8c37a07f127718a0e5229f0ce02185e64d58802dbfddeef29210546dcef89552" exitCode=0
Sep 09 20:16:24.797666 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:24.797313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22" event={"ID":"d496e0d8-645a-4760-8242-3ca54bbcfd00","Type":"ContainerDied","Data":"8c37a07f127718a0e5229f0ce02185e64d58802dbfddeef29210546dcef89552"}
Sep 09 20:16:25.925136 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.925109    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:25.941037 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.940585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7n7b\" (UniqueName: \"kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b\") pod \"d496e0d8-645a-4760-8242-3ca54bbcfd00\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") "
Sep 09 20:16:25.941037 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.940723    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util\") pod \"d496e0d8-645a-4760-8242-3ca54bbcfd00\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") "
Sep 09 20:16:25.941037 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.940753    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle\") pod \"d496e0d8-645a-4760-8242-3ca54bbcfd00\" (UID: \"d496e0d8-645a-4760-8242-3ca54bbcfd00\") "
Sep 09 20:16:25.942010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.941769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle" (OuterVolumeSpecName: "bundle") pod "d496e0d8-645a-4760-8242-3ca54bbcfd00" (UID: "d496e0d8-645a-4760-8242-3ca54bbcfd00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:16:25.943279 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.943251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b" (OuterVolumeSpecName: "kube-api-access-w7n7b") pod "d496e0d8-645a-4760-8242-3ca54bbcfd00" (UID: "d496e0d8-645a-4760-8242-3ca54bbcfd00"). InnerVolumeSpecName "kube-api-access-w7n7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:25.947010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:25.946979    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util" (OuterVolumeSpecName: "util") pod "d496e0d8-645a-4760-8242-3ca54bbcfd00" (UID: "d496e0d8-645a-4760-8242-3ca54bbcfd00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:16:26.041910 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.041873    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-util\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:16:26.041910 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.041903    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d496e0d8-645a-4760-8242-3ca54bbcfd00-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:16:26.041910 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.041915    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7n7b\" (UniqueName: \"kubernetes.io/projected/d496e0d8-645a-4760-8242-3ca54bbcfd00-kube-api-access-w7n7b\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:16:26.807182 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.807144    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22" event={"ID":"d496e0d8-645a-4760-8242-3ca54bbcfd00","Type":"ContainerDied","Data":"56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59"}
Sep 09 20:16:26.807384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.807192    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awrx22"
Sep 09 20:16:26.807384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:26.807196    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c50423f5321bf7706d68737f3faef5b0e58bdae6fca2d3b7a6eca713a2bb59"
Sep 09 20:16:33.581484 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581447    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"]
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="extract"
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581800    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="extract"
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581813    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="pull"
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581819    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="pull"
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581833    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="util"
Sep 09 20:16:33.581871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581839    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="util"
Sep 09 20:16:33.582088 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.581898    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d496e0d8-645a-4760-8242-3ca54bbcfd00" containerName="extract"
Sep 09 20:16:33.585376 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.585359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.587943 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.587916    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:16:33.588097 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.587946    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:16:33.588097 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.587952    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-mhpbj\""
Sep 09 20:16:33.588097 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.588085    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:16:33.595047 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.595017    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"]
Sep 09 20:16:33.595983 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.595962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d3e87591-efad-4a01-879f-bf810d2dc58e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.596102 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.596024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdh5t\" (UniqueName: \"kubernetes.io/projected/d3e87591-efad-4a01-879f-bf810d2dc58e-kube-api-access-pdh5t\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.696745 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.696707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d3e87591-efad-4a01-879f-bf810d2dc58e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.697041 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.696757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdh5t\" (UniqueName: \"kubernetes.io/projected/d3e87591-efad-4a01-879f-bf810d2dc58e-kube-api-access-pdh5t\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.699207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.699185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d3e87591-efad-4a01-879f-bf810d2dc58e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.704588 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.704560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdh5t\" (UniqueName: \"kubernetes.io/projected/d3e87591-efad-4a01-879f-bf810d2dc58e-kube-api-access-pdh5t\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jjdss\" (UID: \"d3e87591-efad-4a01-879f-bf810d2dc58e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:33.896182 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:33.896076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:34.172466 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:34.172440    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"]
Sep 09 20:16:34.175344 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:16:34.175317    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e87591_efad_4a01_879f_bf810d2dc58e.slice/crio-53ccbe3955ad0d055ae8434c8a2ca7de0c7dc46f3793d36573ba958e96833bc5 WatchSource:0}: Error finding container 53ccbe3955ad0d055ae8434c8a2ca7de0c7dc46f3793d36573ba958e96833bc5: Status 404 returned error can't find the container with id 53ccbe3955ad0d055ae8434c8a2ca7de0c7dc46f3793d36573ba958e96833bc5
Sep 09 20:16:34.839805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:34.839767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss" event={"ID":"d3e87591-efad-4a01-879f-bf810d2dc58e","Type":"ContainerStarted","Data":"53ccbe3955ad0d055ae8434c8a2ca7de0c7dc46f3793d36573ba958e96833bc5"}
Sep 09 20:16:37.859320 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:37.859285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss" event={"ID":"d3e87591-efad-4a01-879f-bf810d2dc58e","Type":"ContainerStarted","Data":"54de9bdee88292fac5dd8359852c5cb419547df09fc4f688597d7b29ba2b5909"}
Sep 09 20:16:37.859716 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:37.859365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:16:37.882398 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:37.882345    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss" podStartSLOduration=1.725230951 podStartE2EDuration="4.882325376s" totalImagesPullingTime="3.157094425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:37.880730935 +0000 UTC m=+337.255269029" watchObservedRunningTime="2026-09-09 20:16:37.882325376 +0000 UTC m=+337.256863470"
Sep 09 20:16:38.821179 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.821141    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-p5dcp"]
Sep 09 20:16:38.823547 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.823530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:38.825886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.825855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:16:38.825886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.825874    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jtx85\""
Sep 09 20:16:38.826128 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.825964    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:16:38.832791 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.832759    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-p5dcp"]
Sep 09 20:16:38.844676 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.844639    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxncg\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-kube-api-access-nxncg\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:38.844836 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.844716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:38.945933 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.945903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxncg\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-kube-api-access-nxncg\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:38.946455 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.945956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:38.946455 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:16:38.946157    2564 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 09 20:16:38.946455 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:16:38.946183    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-p5dcp: secret "keda-admission-webhooks-certs" not found
Sep 09 20:16:38.946455 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:16:38.946259    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates podName:a305557f-e678-41fa-ad7e-78bd6e6a5b01 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:39.44622728 +0000 UTC m=+338.820765366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates") pod "keda-admission-6c4d864b86-p5dcp" (UID: "a305557f-e678-41fa-ad7e-78bd6e6a5b01") : secret "keda-admission-webhooks-certs" not found
Sep 09 20:16:38.953958 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:38.953931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxncg\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-kube-api-access-nxncg\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:39.449550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:39.449508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:39.452001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:39.451972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a305557f-e678-41fa-ad7e-78bd6e6a5b01-certificates\") pod \"keda-admission-6c4d864b86-p5dcp\" (UID: \"a305557f-e678-41fa-ad7e-78bd6e6a5b01\") " pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:39.735577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:39.735475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:40.011360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:40.011175    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-p5dcp"]
Sep 09 20:16:40.013869 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:16:40.013843    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda305557f_e678_41fa_ad7e_78bd6e6a5b01.slice/crio-fe74b327a2df8a328029246bb078ca9bb611a88344611c8e338a101b177a3f0a WatchSource:0}: Error finding container fe74b327a2df8a328029246bb078ca9bb611a88344611c8e338a101b177a3f0a: Status 404 returned error can't find the container with id fe74b327a2df8a328029246bb078ca9bb611a88344611c8e338a101b177a3f0a
Sep 09 20:16:40.875018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:40.874981    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp" event={"ID":"a305557f-e678-41fa-ad7e-78bd6e6a5b01","Type":"ContainerStarted","Data":"fe74b327a2df8a328029246bb078ca9bb611a88344611c8e338a101b177a3f0a"}
Sep 09 20:16:41.881117 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:41.881036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp" event={"ID":"a305557f-e678-41fa-ad7e-78bd6e6a5b01","Type":"ContainerStarted","Data":"5c62195e8b536cdcfd5a6b922e0a92e086aa50109be5250dbd229c4f462f2fd6"}
Sep 09 20:16:41.881470 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:41.881183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:16:41.896631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:41.896577    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp" podStartSLOduration=2.310146142 podStartE2EDuration="3.896558448s" totalImagesPullingTime="1.586412306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:41.895611441 +0000 UTC m=+341.270149537" watchObservedRunningTime="2026-09-09 20:16:41.896558448 +0000 UTC m=+341.271096542"
Sep 09 20:16:58.866937 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:16:58.866859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jjdss"
Sep 09 20:17:02.888640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:02.888607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-p5dcp"
Sep 09 20:17:34.438277 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.438242    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f79fb54c7-zmcxj"]
Sep 09 20:17:34.440976 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.440955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.452849 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.452820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f79fb54c7-zmcxj"]
Sep 09 20:17:34.534148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534109    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hkn\" (UniqueName: \"kubernetes.io/projected/b5100a4b-1731-4481-aef5-f9dfa2df9831-kube-api-access-68hkn\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-oauth-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-trusted-ca-bundle\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-oauth-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.534587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.534383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-service-ca\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-oauth-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-service-ca\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hkn\" (UniqueName: \"kubernetes.io/projected/b5100a4b-1731-4481-aef5-f9dfa2df9831-kube-api-access-68hkn\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-oauth-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-trusted-ca-bundle\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.635472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.635438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.636049 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.636011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-oauth-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.636285 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.636260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-service-ca\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.636430 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.636405    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.636564 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.636540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5100a4b-1731-4481-aef5-f9dfa2df9831-trusted-ca-bundle\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.638137 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.638115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-serving-cert\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.638236 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.638159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5100a4b-1731-4481-aef5-f9dfa2df9831-console-oauth-config\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.642916 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.642895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hkn\" (UniqueName: \"kubernetes.io/projected/b5100a4b-1731-4481-aef5-f9dfa2df9831-kube-api-access-68hkn\") pod \"console-f79fb54c7-zmcxj\" (UID: \"b5100a4b-1731-4481-aef5-f9dfa2df9831\") " pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:34.754848 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:34.754747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:35.030716 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:35.030688    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f79fb54c7-zmcxj"]
Sep 09 20:17:35.033104 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:17:35.033074    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5100a4b_1731_4481_aef5_f9dfa2df9831.slice/crio-baf0dd03a1eb86392207828be0412093cf4696a60b2c6fcc7583b785b6463f14 WatchSource:0}: Error finding container baf0dd03a1eb86392207828be0412093cf4696a60b2c6fcc7583b785b6463f14: Status 404 returned error can't find the container with id baf0dd03a1eb86392207828be0412093cf4696a60b2c6fcc7583b785b6463f14
Sep 09 20:17:35.119369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:35.119333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f79fb54c7-zmcxj" event={"ID":"b5100a4b-1731-4481-aef5-f9dfa2df9831","Type":"ContainerStarted","Data":"624d42320bd6d8919e8e145da4b051995f0825934998374ca0f16996680e5385"}
Sep 09 20:17:35.119562 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:35.119379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f79fb54c7-zmcxj" event={"ID":"b5100a4b-1731-4481-aef5-f9dfa2df9831","Type":"ContainerStarted","Data":"baf0dd03a1eb86392207828be0412093cf4696a60b2c6fcc7583b785b6463f14"}
Sep 09 20:17:35.135973 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:35.135911    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f79fb54c7-zmcxj" podStartSLOduration=1.1358929899999999 podStartE2EDuration="1.13589299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:35.135092841 +0000 UTC m=+394.509630935" watchObservedRunningTime="2026-09-09 20:17:35.13589299 +0000 UTC m=+394.510431085"
Sep 09 20:17:44.755607 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:44.755556    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:44.756018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:44.755655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:44.760556 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:44.760531    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:45.166276 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:45.166247    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f79fb54c7-zmcxj"
Sep 09 20:17:45.210726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:45.210686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:17:55.736271 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.736236    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:17:55.739726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.739701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.741985 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.741960    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:17:55.742648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.742630    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:17:55.742648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.742637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:17:55.742803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.742721    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8lz58\""
Sep 09 20:17:55.748953 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.748925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:17:55.776730 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.776695    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tj96n"]
Sep 09 20:17:55.779824 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.779801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:55.782150 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.782127    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2m9cf\""
Sep 09 20:17:55.782276 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.782220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:17:55.787688 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.787656    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tj96n"]
Sep 09 20:17:55.828451 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.828417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.828651 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.828532    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwrz\" (UniqueName: \"kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.929959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.929922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwrz\" (UniqueName: \"kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.930187 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.929975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.930187 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.929999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghpf\" (UniqueName: \"kubernetes.io/projected/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-kube-api-access-6ghpf\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:55.930187 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.930020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-data\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:55.932511 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.932488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:55.938685 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:55.938658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwrz\" (UniqueName: \"kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz\") pod \"kserve-controller-manager-55b9566966-vcjh2\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") " pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:56.030849 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.030818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghpf\" (UniqueName: \"kubernetes.io/projected/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-kube-api-access-6ghpf\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:56.031007 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.030863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-data\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:56.031257 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.031241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-data\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:56.039402 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.039372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghpf\" (UniqueName: \"kubernetes.io/projected/4d2341f2-dcda-4d86-88dd-40d44b7a3a22-kube-api-access-6ghpf\") pod \"seaweedfs-64568bcd49-tj96n\" (UID: \"4d2341f2-dcda-4d86-88dd-40d44b7a3a22\") " pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:56.052810 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.052779    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:17:56.090867 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.090169    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:17:56.327133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.327099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:17:56.329101 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:17:56.329072    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada9a82e_d801_451e_8608_4d19bc69073a.slice/crio-283f4d12b48febbc45ccb5f5e7681ea616d5ed780f539c2d05a013e84d8832d2 WatchSource:0}: Error finding container 283f4d12b48febbc45ccb5f5e7681ea616d5ed780f539c2d05a013e84d8832d2: Status 404 returned error can't find the container with id 283f4d12b48febbc45ccb5f5e7681ea616d5ed780f539c2d05a013e84d8832d2
Sep 09 20:17:56.371122 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:56.371097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tj96n"]
Sep 09 20:17:56.373436 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:17:56.373408    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2341f2_dcda_4d86_88dd_40d44b7a3a22.slice/crio-5bb87a90dab0e1c4f83d0ea9482c232c5eba430f085c0d491b09b96b5bb49bbb WatchSource:0}: Error finding container 5bb87a90dab0e1c4f83d0ea9482c232c5eba430f085c0d491b09b96b5bb49bbb: Status 404 returned error can't find the container with id 5bb87a90dab0e1c4f83d0ea9482c232c5eba430f085c0d491b09b96b5bb49bbb
Sep 09 20:17:57.228668 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:57.228616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" event={"ID":"ada9a82e-d801-451e-8608-4d19bc69073a","Type":"ContainerStarted","Data":"283f4d12b48febbc45ccb5f5e7681ea616d5ed780f539c2d05a013e84d8832d2"}
Sep 09 20:17:57.230195 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:17:57.230159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tj96n" event={"ID":"4d2341f2-dcda-4d86-88dd-40d44b7a3a22","Type":"ContainerStarted","Data":"5bb87a90dab0e1c4f83d0ea9482c232c5eba430f085c0d491b09b96b5bb49bbb"}
Sep 09 20:18:01.252137 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.252090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" event={"ID":"ada9a82e-d801-451e-8608-4d19bc69073a","Type":"ContainerStarted","Data":"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"}
Sep 09 20:18:01.252599 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.252200    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:18:01.253713 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.253688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tj96n" event={"ID":"4d2341f2-dcda-4d86-88dd-40d44b7a3a22","Type":"ContainerStarted","Data":"f0bda1cfd083f1d5913a097fd2367b68af310af9dfd4118fbb3f84d41df7ef91"}
Sep 09 20:18:01.253842 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.253737    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:18:01.269501 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.269443    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" podStartSLOduration=2.406828441 podStartE2EDuration="6.269427092s" totalImagesPullingTime="3.862598651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:01.267207598 +0000 UTC m=+420.641745691" watchObservedRunningTime="2026-09-09 20:18:01.269427092 +0000 UTC m=+420.643965187"
Sep 09 20:18:01.282117 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:01.282032    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tj96n" podStartSLOduration=2.3683016869999998 podStartE2EDuration="6.28201127s" totalImagesPullingTime="3.913709583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:01.281730885 +0000 UTC m=+420.656268979" watchObservedRunningTime="2026-09-09 20:18:01.28201127 +0000 UTC m=+420.656549367"
Sep 09 20:18:07.261196 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:07.261163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tj96n"
Sep 09 20:18:10.236505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.236466    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f5b77db46-jbdzx" podUID="d906771d-4945-4d55-adcd-041d6cfcc898" containerName="console" containerID="cri-o://588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480" gracePeriod=15
Sep 09 20:18:10.484978 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.484953    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:18:10.561903 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.561869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.561903 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.561904    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.561955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.561978    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k89fd\" (UniqueName: \"kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562048    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562109    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert\") pod \"d906771d-4945-4d55-adcd-041d6cfcc898\" (UID: \"d906771d-4945-4d55-adcd-041d6cfcc898\") "
Sep 09 20:18:10.562477 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562450    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config" (OuterVolumeSpecName: "console-config") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:10.562528 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:10.562574 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562517    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca" (OuterVolumeSpecName: "service-ca") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:10.562638 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.562621    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:10.564278 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.564256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:10.564680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.564657    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:10.564746 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.564672    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd" (OuterVolumeSpecName: "kube-api-access-k89fd") pod "d906771d-4945-4d55-adcd-041d6cfcc898" (UID: "d906771d-4945-4d55-adcd-041d6cfcc898"). InnerVolumeSpecName "kube-api-access-k89fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:10.663017 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.662980    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k89fd\" (UniqueName: \"kubernetes.io/projected/d906771d-4945-4d55-adcd-041d6cfcc898-kube-api-access-k89fd\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663017 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663008    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-trusted-ca-bundle\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663017 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663018    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663017 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663026    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-oauth-serving-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663305 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663036    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-service-ca\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663305 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663043    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d906771d-4945-4d55-adcd-041d6cfcc898-console-oauth-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:10.663305 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:10.663051    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d906771d-4945-4d55-adcd-041d6cfcc898-console-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:11.302157 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.302120    2564 generic.go:356] "Generic (PLEG): container finished" podID="d906771d-4945-4d55-adcd-041d6cfcc898" containerID="588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480" exitCode=0
Sep 09 20:18:11.302650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.302169    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f5b77db46-jbdzx"
Sep 09 20:18:11.302650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.302189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b77db46-jbdzx" event={"ID":"d906771d-4945-4d55-adcd-041d6cfcc898","Type":"ContainerDied","Data":"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"}
Sep 09 20:18:11.302650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.302241    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f5b77db46-jbdzx" event={"ID":"d906771d-4945-4d55-adcd-041d6cfcc898","Type":"ContainerDied","Data":"97c93933f86e967e006441774a99e2a799d1f9e11585325a311a52150842552a"}
Sep 09 20:18:11.302650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.302263    2564 scope.go:117] "RemoveContainer" containerID="588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"
Sep 09 20:18:11.310875 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.310855    2564 scope.go:117] "RemoveContainer" containerID="588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"
Sep 09 20:18:11.311186 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:18:11.311168    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480\": container with ID starting with 588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480 not found: ID does not exist" containerID="588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"
Sep 09 20:18:11.311238 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.311196    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480"} err="failed to get container status \"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480\": rpc error: code = NotFound desc = could not find container \"588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480\": container with ID starting with 588fecf9c137a796717ff6100bdea253c076afbfef521399bf168db2505b9480 not found: ID does not exist"
Sep 09 20:18:11.317677 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.317651    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:18:11.322280 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:11.322253    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f5b77db46-jbdzx"]
Sep 09 20:18:13.119694 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:13.119657    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d906771d-4945-4d55-adcd-041d6cfcc898" path="/var/lib/kubelet/pods/d906771d-4945-4d55-adcd-041d6cfcc898/volumes"
Sep 09 20:18:32.263740 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:32.263660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:18:38.078821 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.078777    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:18:38.079422 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.078996    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" podUID="ada9a82e-d801-451e-8608-4d19bc69073a" containerName="manager" containerID="cri-o://fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0" gracePeriod=10
Sep 09 20:18:38.103406 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.103376    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-qzg97"]
Sep 09 20:18:38.103744 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.103731    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d906771d-4945-4d55-adcd-041d6cfcc898" containerName="console"
Sep 09 20:18:38.103788 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.103746    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d906771d-4945-4d55-adcd-041d6cfcc898" containerName="console"
Sep 09 20:18:38.103825 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.103799    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d906771d-4945-4d55-adcd-041d6cfcc898" containerName="console"
Sep 09 20:18:38.111504 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.111478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.115887 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.115858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-qzg97"]
Sep 09 20:18:38.180595 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.180557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pfp\" (UniqueName: \"kubernetes.io/projected/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-kube-api-access-k7pfp\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.180777 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.180627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-cert\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.281724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.281688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pfp\" (UniqueName: \"kubernetes.io/projected/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-kube-api-access-k7pfp\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.281884 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.281743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-cert\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.284110 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.284082    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-cert\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.289243 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.289210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pfp\" (UniqueName: \"kubernetes.io/projected/fab0065b-d50d-40ea-b1c8-b4fc7f0545a0-kube-api-access-k7pfp\") pod \"kserve-controller-manager-55b9566966-qzg97\" (UID: \"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0\") " pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.328355 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.328329    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:18:38.383052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.382961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert\") pod \"ada9a82e-d801-451e-8608-4d19bc69073a\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") "
Sep 09 20:18:38.383052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.383052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crwrz\" (UniqueName: \"kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz\") pod \"ada9a82e-d801-451e-8608-4d19bc69073a\" (UID: \"ada9a82e-d801-451e-8608-4d19bc69073a\") "
Sep 09 20:18:38.385263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.385225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert" (OuterVolumeSpecName: "cert") pod "ada9a82e-d801-451e-8608-4d19bc69073a" (UID: "ada9a82e-d801-451e-8608-4d19bc69073a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:38.385390 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.385319    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz" (OuterVolumeSpecName: "kube-api-access-crwrz") pod "ada9a82e-d801-451e-8608-4d19bc69073a" (UID: "ada9a82e-d801-451e-8608-4d19bc69073a"). InnerVolumeSpecName "kube-api-access-crwrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:38.426224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.426182    2564 generic.go:356] "Generic (PLEG): container finished" podID="ada9a82e-d801-451e-8608-4d19bc69073a" containerID="fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0" exitCode=0
Sep 09 20:18:38.426224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.426227    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-vcjh2"
Sep 09 20:18:38.426426 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.426268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" event={"ID":"ada9a82e-d801-451e-8608-4d19bc69073a","Type":"ContainerDied","Data":"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"}
Sep 09 20:18:38.426426 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.426318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-vcjh2" event={"ID":"ada9a82e-d801-451e-8608-4d19bc69073a","Type":"ContainerDied","Data":"283f4d12b48febbc45ccb5f5e7681ea616d5ed780f539c2d05a013e84d8832d2"}
Sep 09 20:18:38.426426 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.426340    2564 scope.go:117] "RemoveContainer" containerID="fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"
Sep 09 20:18:38.435607 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.435588    2564 scope.go:117] "RemoveContainer" containerID="fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"
Sep 09 20:18:38.435897 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:18:38.435879    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0\": container with ID starting with fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0 not found: ID does not exist" containerID="fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"
Sep 09 20:18:38.435966 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.435904    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0"} err="failed to get container status \"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0\": rpc error: code = NotFound desc = could not find container \"fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0\": container with ID starting with fbb0964a543ccd6e5c78c3668d26969846f054be1b1bee912bb490b13e8809d0 not found: ID does not exist"
Sep 09 20:18:38.446476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.446448    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:18:38.449985 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.449962    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-vcjh2"]
Sep 09 20:18:38.460566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.460540    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:38.483746 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.483718    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwrz\" (UniqueName: \"kubernetes.io/projected/ada9a82e-d801-451e-8608-4d19bc69073a-kube-api-access-crwrz\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:38.483746 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.483744    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ada9a82e-d801-451e-8608-4d19bc69073a-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:18:38.729436 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:38.729412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-qzg97"]
Sep 09 20:18:38.731699 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:18:38.731661    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab0065b_d50d_40ea_b1c8_b4fc7f0545a0.slice/crio-21962ae138d0d34cb0d8cf5c59309b4976228944923ef06d2a00f0888260902b WatchSource:0}: Error finding container 21962ae138d0d34cb0d8cf5c59309b4976228944923ef06d2a00f0888260902b: Status 404 returned error can't find the container with id 21962ae138d0d34cb0d8cf5c59309b4976228944923ef06d2a00f0888260902b
Sep 09 20:18:39.119437 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:39.119405    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada9a82e-d801-451e-8608-4d19bc69073a" path="/var/lib/kubelet/pods/ada9a82e-d801-451e-8608-4d19bc69073a/volumes"
Sep 09 20:18:39.432030 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:39.431930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-qzg97" event={"ID":"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0","Type":"ContainerStarted","Data":"7b53e9deb83362fb4434abfa0147ad1a4d8a8a12361914a9276b9a006b1f5453"}
Sep 09 20:18:39.432030 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:39.431989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-qzg97" event={"ID":"fab0065b-d50d-40ea-b1c8-b4fc7f0545a0","Type":"ContainerStarted","Data":"21962ae138d0d34cb0d8cf5c59309b4976228944923ef06d2a00f0888260902b"}
Sep 09 20:18:39.432299 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:39.432115    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:18:39.448551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:18:39.448497    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-qzg97" podStartSLOduration=1.024319092 podStartE2EDuration="1.44848102s" totalImagesPullingTime="424.161928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:39.446615819 +0000 UTC m=+458.821153915" watchObservedRunningTime="2026-09-09 20:18:39.44848102 +0000 UTC m=+458.823019114"
Sep 09 20:19:10.442311 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:10.442269    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-qzg97"
Sep 09 20:19:11.971650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.971615    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-hpswd"]
Sep 09 20:19:11.972044 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.971975    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada9a82e-d801-451e-8608-4d19bc69073a" containerName="manager"
Sep 09 20:19:11.972044 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.971988    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada9a82e-d801-451e-8608-4d19bc69073a" containerName="manager"
Sep 09 20:19:11.972148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.972071    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada9a82e-d801-451e-8608-4d19bc69073a" containerName="manager"
Sep 09 20:19:11.976453 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.976430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:11.979760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.979690    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2zfjh\""
Sep 09 20:19:11.979962 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.979938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:19:11.982726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.982701    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-25f7c"]
Sep 09 20:19:11.986400 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.986377    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hpswd"]
Sep 09 20:19:11.986507 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.986474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:11.988735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.988713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:19:11.988898 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.988780    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:19:11.989008 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.988830    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-25qxc\""
Sep 09 20:19:11.999152 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:11.999115    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-25f7c"]
Sep 09 20:19:12.072130 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.072052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrpt\" (UniqueName: \"kubernetes.io/projected/f9b2c9a9-a235-4f10-b68c-c84b6633d331-kube-api-access-dwrpt\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.072341 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.072138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbggc\" (UniqueName: \"kubernetes.io/projected/48cd2969-5c95-459a-8700-fc627f7c6ddc-kube-api-access-jbggc\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.072341 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.072167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cd2969-5c95-459a-8700-fc627f7c6ddc-tls-certs\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.072341 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.072201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-cert\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.072341 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.072287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.172612 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.172574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.172612 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.172617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrpt\" (UniqueName: \"kubernetes.io/projected/f9b2c9a9-a235-4f10-b68c-c84b6633d331-kube-api-access-dwrpt\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.172883 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:19:12.172723    2564 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 20:19:12.172883 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.172779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbggc\" (UniqueName: \"kubernetes.io/projected/48cd2969-5c95-459a-8700-fc627f7c6ddc-kube-api-access-jbggc\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.172883 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:19:12.172798    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs podName:f9b2c9a9-a235-4f10-b68c-c84b6633d331 nodeName:}" failed. No retries permitted until 2026-09-09 20:19:12.672775094 +0000 UTC m=+492.047313179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs") pod "odh-model-controller-7b966d7984-25f7c" (UID: "f9b2c9a9-a235-4f10-b68c-c84b6633d331") : secret "odh-model-controller-metrics-tls" not found
Sep 09 20:19:12.172883 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.172858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cd2969-5c95-459a-8700-fc627f7c6ddc-tls-certs\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.173242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.172934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-cert\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.175393 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.175369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48cd2969-5c95-459a-8700-fc627f7c6ddc-tls-certs\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.175516 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.175372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-cert\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.183452 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.183430    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbggc\" (UniqueName: \"kubernetes.io/projected/48cd2969-5c95-459a-8700-fc627f7c6ddc-kube-api-access-jbggc\") pod \"model-serving-api-7b954ffc95-hpswd\" (UID: \"48cd2969-5c95-459a-8700-fc627f7c6ddc\") " pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.183572 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.183561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrpt\" (UniqueName: \"kubernetes.io/projected/f9b2c9a9-a235-4f10-b68c-c84b6633d331-kube-api-access-dwrpt\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.289655 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.289626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:12.561273 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.561248    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hpswd"]
Sep 09 20:19:12.563324 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:19:12.563296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cd2969_5c95_459a_8700_fc627f7c6ddc.slice/crio-4acef1f09617e2e36166bd96d63920bba55671771d5ebaa60b659f530a6f349f WatchSource:0}: Error finding container 4acef1f09617e2e36166bd96d63920bba55671771d5ebaa60b659f530a6f349f: Status 404 returned error can't find the container with id 4acef1f09617e2e36166bd96d63920bba55671771d5ebaa60b659f530a6f349f
Sep 09 20:19:12.582949 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.582919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hpswd" event={"ID":"48cd2969-5c95-459a-8700-fc627f7c6ddc","Type":"ContainerStarted","Data":"4acef1f09617e2e36166bd96d63920bba55671771d5ebaa60b659f530a6f349f"}
Sep 09 20:19:12.678586 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.678534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.681033 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.681005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2c9a9-a235-4f10-b68c-c84b6633d331-metrics-certs\") pod \"odh-model-controller-7b966d7984-25f7c\" (UID: \"f9b2c9a9-a235-4f10-b68c-c84b6633d331\") " pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:12.898566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:12.898469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:13.172052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:13.171962    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-25f7c"]
Sep 09 20:19:13.175622 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:19:13.175589    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b2c9a9_a235_4f10_b68c_c84b6633d331.slice/crio-354a7817377c37350676e9956b4eb8aa9fc10ebaab1df049ee0a9a260744bff4 WatchSource:0}: Error finding container 354a7817377c37350676e9956b4eb8aa9fc10ebaab1df049ee0a9a260744bff4: Status 404 returned error can't find the container with id 354a7817377c37350676e9956b4eb8aa9fc10ebaab1df049ee0a9a260744bff4
Sep 09 20:19:13.590860 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:13.590815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-25f7c" event={"ID":"f9b2c9a9-a235-4f10-b68c-c84b6633d331","Type":"ContainerStarted","Data":"354a7817377c37350676e9956b4eb8aa9fc10ebaab1df049ee0a9a260744bff4"}
Sep 09 20:19:14.600206 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:14.600164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hpswd" event={"ID":"48cd2969-5c95-459a-8700-fc627f7c6ddc","Type":"ContainerStarted","Data":"b7a1b43a4381deae7ef899ab35996de858424eeffa9ded5f0cda44c218e62d28"}
Sep 09 20:19:14.600693 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:14.600417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:14.615607 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:14.615181    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-hpswd" podStartSLOduration=2.059867524 podStartE2EDuration="3.615159739s" totalImagesPullingTime="1.555292215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:14.61462493 +0000 UTC m=+493.989163038" watchObservedRunningTime="2026-09-09 20:19:14.615159739 +0000 UTC m=+493.989697832"
Sep 09 20:19:16.612464 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:16.612431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-25f7c" event={"ID":"f9b2c9a9-a235-4f10-b68c-c84b6633d331","Type":"ContainerStarted","Data":"8f2bccdd57a7599cb52421f4fd5dc474906f0fb62457cebbdb99a007a65e47b8"}
Sep 09 20:19:16.612956 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:16.612559    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:16.627024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:16.626972    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-25f7c" podStartSLOduration=2.900815032 podStartE2EDuration="5.626951499s" totalImagesPullingTime="2.726136467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:16.625835519 +0000 UTC m=+496.000373614" watchObservedRunningTime="2026-09-09 20:19:16.626951499 +0000 UTC m=+496.001489592"
Sep 09 20:19:25.610448 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:25.610417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-hpswd"
Sep 09 20:19:27.619407 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:27.619379    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-25f7c"
Sep 09 20:19:28.396265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.396227    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-87kfr"]
Sep 09 20:19:28.399567 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.399550    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.401548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.401531    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:19:28.407023 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.406923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-87kfr"]
Sep 09 20:19:28.526301 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.526260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfjl\" (UniqueName: \"kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.526486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.526313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.627420 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.627376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfjl\" (UniqueName: \"kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.627829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.627431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.627870 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.627828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.635011 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.634978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfjl\" (UniqueName: \"kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl\") pod \"s3-init-87kfr\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") " pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.719378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.719285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87kfr"
Sep 09 20:19:28.990846 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:28.990818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-87kfr"]
Sep 09 20:19:28.993398 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:19:28.993368    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c530915_b152_4042_ab5e_cd2d735b7fb8.slice/crio-63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856 WatchSource:0}: Error finding container 63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856: Status 404 returned error can't find the container with id 63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856
Sep 09 20:19:29.673637 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:29.673601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerStarted","Data":"63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856"}
Sep 09 20:19:33.694225 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:33.694190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerStarted","Data":"e99c6fa122438e2b739249242f9a546aab3b06e2c20d2edfa466bc391c4eaa72"}
Sep 09 20:19:36.711864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:36.711826    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerID="e99c6fa122438e2b739249242f9a546aab3b06e2c20d2edfa466bc391c4eaa72" exitCode=0
Sep 09 20:19:36.712386 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:36.711870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerDied","Data":"e99c6fa122438e2b739249242f9a546aab3b06e2c20d2edfa466bc391c4eaa72"}
Sep 09 20:19:41.739854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:41.739821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerStarted","Data":"a5da662d4fb7bef120bb4f91da027a41b97af0be30c4e7f3a416175235a16419"}
Sep 09 20:19:41.757439 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:19:41.757380    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-87kfr" podStartSLOduration=2.7374089230000003 podStartE2EDuration="13.757362619s" totalImagesPullingTime="8.91506744s" totalInitContainerRuntime="2.104886256s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:41.755286204 +0000 UTC m=+521.129824298" watchObservedRunningTime="2026-09-09 20:19:41.757362619 +0000 UTC m=+521.131900712"
Sep 09 20:20:47.039614 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:47.039579    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerID="a5da662d4fb7bef120bb4f91da027a41b97af0be30c4e7f3a416175235a16419" exitCode=0
Sep 09 20:20:47.040023 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:47.039646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerDied","Data":"a5da662d4fb7bef120bb4f91da027a41b97af0be30c4e7f3a416175235a16419"}
Sep 09 20:20:48.184971 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.184941    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87kfr"
Sep 09 20:20:48.316609 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.316519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlfjl\" (UniqueName: \"kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl\") pod \"7c530915-b152-4042-ab5e-cd2d735b7fb8\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") "
Sep 09 20:20:48.316609 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.316567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache\") pod \"7c530915-b152-4042-ab5e-cd2d735b7fb8\" (UID: \"7c530915-b152-4042-ab5e-cd2d735b7fb8\") "
Sep 09 20:20:48.318633 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.318604    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl" (OuterVolumeSpecName: "kube-api-access-qlfjl") pod "7c530915-b152-4042-ab5e-cd2d735b7fb8" (UID: "7c530915-b152-4042-ab5e-cd2d735b7fb8"). InnerVolumeSpecName "kube-api-access-qlfjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:48.335324 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.335282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c530915-b152-4042-ab5e-cd2d735b7fb8" (UID: "7c530915-b152-4042-ab5e-cd2d735b7fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:48.417355 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.417300    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfjl\" (UniqueName: \"kubernetes.io/projected/7c530915-b152-4042-ab5e-cd2d735b7fb8-kube-api-access-qlfjl\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:20:48.417355 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:48.417348    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c530915-b152-4042-ab5e-cd2d735b7fb8-model-cache\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:20:49.050459 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:49.050433    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-87kfr"
Sep 09 20:20:49.050659 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:49.050465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-87kfr" event={"ID":"7c530915-b152-4042-ab5e-cd2d735b7fb8","Type":"ContainerDied","Data":"63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856"}
Sep 09 20:20:49.050659 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:20:49.050501    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c4a6aae604f00b68fc8ca5ff1bb652893bb1aae311c66e4d50c96892f12856"
Sep 09 20:21:00.671379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671346    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:21:00.671869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671737    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerName="download-hf-model"
Sep 09 20:21:00.671869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671750    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerName="download-hf-model"
Sep 09 20:21:00.671869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671765    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerName="s3-init"
Sep 09 20:21:00.671869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerName="s3-init"
Sep 09 20:21:00.671869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.671830    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c530915-b152-4042-ab5e-cd2d735b7fb8" containerName="s3-init"
Sep 09 20:21:00.675040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.675022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.677516 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677494    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 20:21:00.677687 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677591    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:21:00.677687 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677591    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:21:00.677856 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677841    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:21:00.677917 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:21:00.677997 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.677864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:21:00.678165 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.678150    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:21:00.685430 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.685405    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:21:00.721672 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.721642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.721832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.721679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.721832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.721716    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4j47\" (UniqueName: \"kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.721832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.721819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.721980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.721878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.780763 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.780734    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:21:00.784550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.784529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.786390 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.786372    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:21:00.795526 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.795499    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:21:00.822295 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95k67\" (UniqueName: \"kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.822444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.822444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.822444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.822444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4j47\" (UniqueName: \"kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.822444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.822714 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.822714 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.822833 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.822833 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.822833 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.822813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.823242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.823210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.823374 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.823294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.825209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.825190    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.831914 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.831882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4j47\" (UniqueName: \"kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47\") pod \"isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:00.878794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.878759    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:21:00.883867 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.883840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.886286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.886258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-43f1f-predictor-serving-cert\""
Sep 09 20:21:00.886286 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.886279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-43f1f-kube-rbac-proxy-sar-config\""
Sep 09 20:21:00.896619 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.896591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:21:00.923540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.923540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.923726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.923768 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.923811 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m8s\" (UniqueName: \"kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.923856 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.923902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.923945 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.923945 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:00.924074 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923972    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.924074 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.923975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95k67\" (UniqueName: \"kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.924353 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.924328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.924487 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.924468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.926171 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.926143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.931285 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.931262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95k67\" (UniqueName: \"kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67\") pod \"isvc-canary-rollback-predictor-6477dbf789-pd99g\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:00.988642 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.988606    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:21:00.997367 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:00.997344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:01.024782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.024782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m8s\" (UniqueName: \"kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.024782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.024782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.024782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.025185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.024952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.025380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.025347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.026523 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.026449    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-43f1f-kube-rbac-proxy-sar-config\""
Sep 09 20:21:01.026639 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.026527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-43f1f-predictor-serving-cert\""
Sep 09 20:21:01.033030 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.033002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m8s\" (UniqueName: \"kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.034879 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:21:01.034848    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-43f1f-predictor-serving-cert" not found
Sep 09 20:21:01.034991 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:21:01.034929    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls podName:974a1497-4814-47ee-92dc-ad0c1322ebca nodeName:}" failed. No retries permitted until 2026-09-09 20:21:01.534908509 +0000 UTC m=+600.909446594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls") pod "isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca") : secret "isvc-raw-sklearn-batcher-43f1f-predictor-serving-cert" not found
Sep 09 20:21:01.035096 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.035047    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.038571 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.038546    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:21:01.038730 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.038715    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:21:01.046414 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.046322    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:21:01.046733 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.046707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:21:01.097734 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.097701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:01.286228 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.286018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:21:01.288879 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:21:01.288855    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709d4f83_4488_4daf_855c_40523b4f14dd.slice/crio-bcc99da5f85a47b812e345672021d4b31f7b88d673b627e3c8affbd5c2825cb9 WatchSource:0}: Error finding container bcc99da5f85a47b812e345672021d4b31f7b88d673b627e3c8affbd5c2825cb9: Status 404 returned error can't find the container with id bcc99da5f85a47b812e345672021d4b31f7b88d673b627e3c8affbd5c2825cb9
Sep 09 20:21:01.293133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.293113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:21:01.375879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.375845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:21:01.379900 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:21:01.379872    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1dba65_9732_4d63_88cf_177ecd33c763.slice/crio-8ffadb147058128c7349007ce764c6822903ea53fd50fd6a7a3ca6aa5291203d WatchSource:0}: Error finding container 8ffadb147058128c7349007ce764c6822903ea53fd50fd6a7a3ca6aa5291203d: Status 404 returned error can't find the container with id 8ffadb147058128c7349007ce764c6822903ea53fd50fd6a7a3ca6aa5291203d
Sep 09 20:21:01.630794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.630679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.633355 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.633317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") pod \"isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:01.796375 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:01.796331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:02.083241 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.083120    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:21:02.085891 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:21:02.085859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974a1497_4814_47ee_92dc_ad0c1322ebca.slice/crio-7d793707362f90ea6004afff1cadd44d1ab0b07475055b0048c484bd3d416427 WatchSource:0}: Error finding container 7d793707362f90ea6004afff1cadd44d1ab0b07475055b0048c484bd3d416427: Status 404 returned error can't find the container with id 7d793707362f90ea6004afff1cadd44d1ab0b07475055b0048c484bd3d416427
Sep 09 20:21:02.113988 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.113957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerStarted","Data":"2efbba342e1d2c708db9cb8331348c5402cbbdd7267a296cc016d4dfc928e096"}
Sep 09 20:21:02.114686 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.114098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerStarted","Data":"bcc99da5f85a47b812e345672021d4b31f7b88d673b627e3c8affbd5c2825cb9"}
Sep 09 20:21:02.116131 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.116099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerStarted","Data":"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"}
Sep 09 20:21:02.116288 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.116169    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerStarted","Data":"8ffadb147058128c7349007ce764c6822903ea53fd50fd6a7a3ca6aa5291203d"}
Sep 09 20:21:02.118370 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:02.118349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerStarted","Data":"7d793707362f90ea6004afff1cadd44d1ab0b07475055b0048c484bd3d416427"}
Sep 09 20:21:03.125587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:03.125545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerStarted","Data":"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"}
Sep 09 20:21:06.142314 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.142281    2564 generic.go:356] "Generic (PLEG): container finished" podID="709d4f83-4488-4daf-855c-40523b4f14dd" containerID="2efbba342e1d2c708db9cb8331348c5402cbbdd7267a296cc016d4dfc928e096" exitCode=0
Sep 09 20:21:06.142764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.142356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerDied","Data":"2efbba342e1d2c708db9cb8331348c5402cbbdd7267a296cc016d4dfc928e096"}
Sep 09 20:21:06.144317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.144293    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerID="8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2" exitCode=0
Sep 09 20:21:06.144396 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.144377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerDied","Data":"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"}
Sep 09 20:21:06.146391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.146318    2564 generic.go:356] "Generic (PLEG): container finished" podID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerID="54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb" exitCode=0
Sep 09 20:21:06.146391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:06.146367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerDied","Data":"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"}
Sep 09 20:21:27.365270 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:27.365229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerStarted","Data":"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"}
Sep 09 20:21:27.411843 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:27.411819    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:21:28.378687 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:28.378572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerStarted","Data":"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"}
Sep 09 20:21:28.383914 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:28.383812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerStarted","Data":"06aa3ec9a2838a23a871dbe21d26f6d6de14c22f1a5ca16b7c92ea2eadde51a8"}
Sep 09 20:21:30.400225 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.400189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerStarted","Data":"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"}
Sep 09 20:21:30.400633 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.400271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:30.400633 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.400405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:30.402133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.402104    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:21:30.402591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.402569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerStarted","Data":"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"}
Sep 09 20:21:30.404795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.404775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerStarted","Data":"901b25c05d30081f3e278182b9f9cf8e88740fad76348d38d19c928ed6ecf9cf"}
Sep 09 20:21:30.404920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.404906    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:30.419239 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.419195    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podStartSLOduration=2.5942880710000003 podStartE2EDuration="30.419179124s" totalImagesPullingTime="23.970544697s" totalInitContainerRuntime="3.854346356s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:30.416225198 +0000 UTC m=+629.790763304" watchObservedRunningTime="2026-09-09 20:21:30.419179124 +0000 UTC m=+629.793717217"
Sep 09 20:21:30.434331 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:30.434277    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podStartSLOduration=2.597620201 podStartE2EDuration="30.434259287s" totalImagesPullingTime="24.035044062s" totalInitContainerRuntime="3.801595024s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:30.432352314 +0000 UTC m=+629.806890408" watchObservedRunningTime="2026-09-09 20:21:30.434259287 +0000 UTC m=+629.808797440"
Sep 09 20:21:31.411139 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:31.411087    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:21:31.411553 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:31.411108    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:31.412301 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:31.412268    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:21:32.416776 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:32.416739    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:21:33.423972 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:33.423937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerStarted","Data":"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"}
Sep 09 20:21:33.424467 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:33.424214    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:33.424467 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:33.424334    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:33.425696 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:33.425668    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:33.446752 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:33.446674    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podStartSLOduration=3.419774518 podStartE2EDuration="33.446657434s" totalImagesPullingTime="26.12562055s" totalInitContainerRuntime="3.901262366s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:33.444633891 +0000 UTC m=+632.819171990" watchObservedRunningTime="2026-09-09 20:21:33.446657434 +0000 UTC m=+632.821195528"
Sep 09 20:21:34.428841 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:34.428790    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:34.429315 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:34.428813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:34.429848 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:34.429823    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:35.433362 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:35.433320    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:35.433819 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:35.433756    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:35.437164 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:35.437140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:21:36.415878 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:36.415848    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:21:36.416474 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:36.416448    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:21:36.439510 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:36.439463    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:36.440000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:36.439810    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:37.421433 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:37.421399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:21:37.421879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:37.421856    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:21:46.417020 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:46.416980    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:21:46.440054 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:46.440006    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:46.440606 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:46.440569    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:47.422726 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:47.422691    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:21:56.417323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:56.417278    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:21:56.440113 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:56.440041    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:21:56.440533 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:56.440503    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:57.422025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:21:57.421980    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:22:06.416476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:06.416435    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:22:06.439879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:06.439837    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:22:06.440443 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:06.440403    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:07.422318 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:07.422279    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:22:16.416660 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:16.416619    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:22:16.439857 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:16.439810    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:22:16.440478 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:16.440438    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:17.422544 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:17.422508    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:22:26.416681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:26.416641    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:22:26.440147 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:26.440096    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:22:26.440637 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:26.440612    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:27.421982 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:27.421944    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:22:36.416629 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:36.416589    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:22:36.440042 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:36.439995    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:22:36.440495 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:36.440464    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:37.422084 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:37.422022    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:22:46.113912 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:46.113873    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:22:46.114399 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:46.114246    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:46.417916 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:46.417834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:22:47.422474 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.422443    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:22:47.533877 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.533839    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:22:47.534164 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.534137    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container" containerID="cri-o://1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779" gracePeriod=30
Sep 09 20:22:47.534258 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.534173    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kube-rbac-proxy" containerID="cri-o://9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c" gracePeriod=30
Sep 09 20:22:47.811976 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.811945    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerID="9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c" exitCode=2
Sep 09 20:22:47.812174 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:47.812025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerDied","Data":"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"}
Sep 09 20:22:50.563965 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.563932    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:22:50.564457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.564193    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container" containerID="cri-o://06aa3ec9a2838a23a871dbe21d26f6d6de14c22f1a5ca16b7c92ea2eadde51a8" gracePeriod=30
Sep 09 20:22:50.564457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.564294    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kube-rbac-proxy" containerID="cri-o://901b25c05d30081f3e278182b9f9cf8e88740fad76348d38d19c928ed6ecf9cf" gracePeriod=30
Sep 09 20:22:50.642801 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.642766    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:22:50.646650 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.646624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.655308 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.655282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:22:50.727119 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.727047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.727307 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.727168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.727307 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.727233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.727307 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.727266    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggngk\" (UniqueName: \"kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.727472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.727326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.827881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.827779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828048 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.827879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828048 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.827923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828048 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.827952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828048 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.827983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggngk\" (UniqueName: \"kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828529 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.828503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828802 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.828782    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.828934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.828792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.829991 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.829969    2564 generic.go:356] "Generic (PLEG): container finished" podID="709d4f83-4488-4daf-855c-40523b4f14dd" containerID="901b25c05d30081f3e278182b9f9cf8e88740fad76348d38d19c928ed6ecf9cf" exitCode=2
Sep 09 20:22:50.830103 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.830037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerDied","Data":"901b25c05d30081f3e278182b9f9cf8e88740fad76348d38d19c928ed6ecf9cf"}
Sep 09 20:22:50.830606 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.830586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.835549 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.835524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggngk\" (UniqueName: \"kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk\") pod \"isvc-canary-rollback-predictor-6477dbf789-t6cq7\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:50.957627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:50.957583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:51.236272 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:51.236183    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:22:51.238969 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:22:51.238936    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b019bf_ddeb_4378_a8d7_a12d493dc9fe.slice/crio-99a0e10072500ae3cd3f4f9942de107ea0b00d35a870dd1715c123206dc1568a WatchSource:0}: Error finding container 99a0e10072500ae3cd3f4f9942de107ea0b00d35a870dd1715c123206dc1568a: Status 404 returned error can't find the container with id 99a0e10072500ae3cd3f4f9942de107ea0b00d35a870dd1715c123206dc1568a
Sep 09 20:22:51.411281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:51.411236    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:22:51.839232 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:51.839185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerStarted","Data":"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"}
Sep 09 20:22:51.839763 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:51.839265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerStarted","Data":"99a0e10072500ae3cd3f4f9942de107ea0b00d35a870dd1715c123206dc1568a"}
Sep 09 20:22:52.417335 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.417296    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 09 20:22:52.782550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.782524    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:22:52.843497 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.843464    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location\") pod \"8c1dba65-9732-4d63-88cf-177ecd33c763\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") "
Sep 09 20:22:52.843948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.843513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls\") pod \"8c1dba65-9732-4d63-88cf-177ecd33c763\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") "
Sep 09 20:22:52.843948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.843539    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config\") pod \"8c1dba65-9732-4d63-88cf-177ecd33c763\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") "
Sep 09 20:22:52.843948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.843607    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95k67\" (UniqueName: \"kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67\") pod \"8c1dba65-9732-4d63-88cf-177ecd33c763\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") "
Sep 09 20:22:52.843948 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.843630    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert\") pod \"8c1dba65-9732-4d63-88cf-177ecd33c763\" (UID: \"8c1dba65-9732-4d63-88cf-177ecd33c763\") "
Sep 09 20:22:52.847155 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.844309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c1dba65-9732-4d63-88cf-177ecd33c763" (UID: "8c1dba65-9732-4d63-88cf-177ecd33c763"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:52.847155 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.845229    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c1dba65-9732-4d63-88cf-177ecd33c763" (UID: "8c1dba65-9732-4d63-88cf-177ecd33c763"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:52.847155 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.845256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c1dba65-9732-4d63-88cf-177ecd33c763" (UID: "8c1dba65-9732-4d63-88cf-177ecd33c763"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:52.848434 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.847464    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67" (OuterVolumeSpecName: "kube-api-access-95k67") pod "8c1dba65-9732-4d63-88cf-177ecd33c763" (UID: "8c1dba65-9732-4d63-88cf-177ecd33c763"). InnerVolumeSpecName "kube-api-access-95k67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:52.852639 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.852606    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c1dba65-9732-4d63-88cf-177ecd33c763" (UID: "8c1dba65-9732-4d63-88cf-177ecd33c763"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:52.854926 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.854898    2564 generic.go:356] "Generic (PLEG): container finished" podID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerID="1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779" exitCode=0
Sep 09 20:22:52.855040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.854986    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"
Sep 09 20:22:52.855040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.855030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerDied","Data":"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"}
Sep 09 20:22:52.855181 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.855138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g" event={"ID":"8c1dba65-9732-4d63-88cf-177ecd33c763","Type":"ContainerDied","Data":"8ffadb147058128c7349007ce764c6822903ea53fd50fd6a7a3ca6aa5291203d"}
Sep 09 20:22:52.855181 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.855174    2564 scope.go:117] "RemoveContainer" containerID="9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"
Sep 09 20:22:52.866818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.866796    2564 scope.go:117] "RemoveContainer" containerID="1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"
Sep 09 20:22:52.875326 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.875294    2564 scope.go:117] "RemoveContainer" containerID="8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"
Sep 09 20:22:52.878296 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.878266    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:22:52.881622 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.881600    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-pd99g"]
Sep 09 20:22:52.884955 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.884934    2564 scope.go:117] "RemoveContainer" containerID="9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"
Sep 09 20:22:52.885300 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:22:52.885278    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c\": container with ID starting with 9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c not found: ID does not exist" containerID="9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"
Sep 09 20:22:52.885382 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.885312    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c"} err="failed to get container status \"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c\": rpc error: code = NotFound desc = could not find container \"9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c\": container with ID starting with 9e5dbaa14da164c23b246febb5ef22fd1f50cd2bf0a9ffea69acba4a9171792c not found: ID does not exist"
Sep 09 20:22:52.885382 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.885338    2564 scope.go:117] "RemoveContainer" containerID="1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"
Sep 09 20:22:52.885643 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:22:52.885624    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779\": container with ID starting with 1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779 not found: ID does not exist" containerID="1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"
Sep 09 20:22:52.885684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.885648    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779"} err="failed to get container status \"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779\": rpc error: code = NotFound desc = could not find container \"1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779\": container with ID starting with 1e47e8284ba177d820c8109a5525c48f24fe47c3d0fdc2f4e410231a10875779 not found: ID does not exist"
Sep 09 20:22:52.885684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.885662    2564 scope.go:117] "RemoveContainer" containerID="8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"
Sep 09 20:22:52.885959 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:22:52.885941    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2\": container with ID starting with 8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2 not found: ID does not exist" containerID="8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"
Sep 09 20:22:52.886010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.885965    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2"} err="failed to get container status \"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2\": rpc error: code = NotFound desc = could not find container \"8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2\": container with ID starting with 8ab90cc93b2b60ed5adc0ffabf0aa16e3ee7e1579e5e0a9db5f647631b0fe8b2 not found: ID does not exist"
Sep 09 20:22:52.945360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.945323    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95k67\" (UniqueName: \"kubernetes.io/projected/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-api-access-95k67\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:52.945360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.945357    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:52.945640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.945372    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c1dba65-9732-4d63-88cf-177ecd33c763-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:52.945640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.945385    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c1dba65-9732-4d63-88cf-177ecd33c763-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:52.945640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:52.945398    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c1dba65-9732-4d63-88cf-177ecd33c763-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:53.119192 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:53.119162    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" path="/var/lib/kubelet/pods/8c1dba65-9732-4d63-88cf-177ecd33c763/volumes"
Sep 09 20:22:55.872777 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.872742    2564 generic.go:356] "Generic (PLEG): container finished" podID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerID="4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1" exitCode=0
Sep 09 20:22:55.873218 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.872808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerDied","Data":"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"}
Sep 09 20:22:55.875510 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.875453    2564 generic.go:356] "Generic (PLEG): container finished" podID="709d4f83-4488-4daf-855c-40523b4f14dd" containerID="06aa3ec9a2838a23a871dbe21d26f6d6de14c22f1a5ca16b7c92ea2eadde51a8" exitCode=0
Sep 09 20:22:55.875510 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.875483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerDied","Data":"06aa3ec9a2838a23a871dbe21d26f6d6de14c22f1a5ca16b7c92ea2eadde51a8"}
Sep 09 20:22:55.926186 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.926158    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:22:55.967453 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.967415    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location\") pod \"709d4f83-4488-4daf-855c-40523b4f14dd\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") "
Sep 09 20:22:55.967596 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.967475    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4j47\" (UniqueName: \"kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47\") pod \"709d4f83-4488-4daf-855c-40523b4f14dd\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") "
Sep 09 20:22:55.967596 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.967502    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls\") pod \"709d4f83-4488-4daf-855c-40523b4f14dd\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") "
Sep 09 20:22:55.967857 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.967829    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "709d4f83-4488-4daf-855c-40523b4f14dd" (UID: "709d4f83-4488-4daf-855c-40523b4f14dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:55.970027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.969990    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47" (OuterVolumeSpecName: "kube-api-access-r4j47") pod "709d4f83-4488-4daf-855c-40523b4f14dd" (UID: "709d4f83-4488-4daf-855c-40523b4f14dd"). InnerVolumeSpecName "kube-api-access-r4j47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:55.970027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:55.970004    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "709d4f83-4488-4daf-855c-40523b4f14dd" (UID: "709d4f83-4488-4daf-855c-40523b4f14dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:56.068619 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config\") pod \"709d4f83-4488-4daf-855c-40523b4f14dd\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") "
Sep 09 20:22:56.068794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068629    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert\") pod \"709d4f83-4488-4daf-855c-40523b4f14dd\" (UID: \"709d4f83-4488-4daf-855c-40523b4f14dd\") "
Sep 09 20:22:56.068896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068878    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/709d4f83-4488-4daf-855c-40523b4f14dd-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:56.068994 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068905    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4j47\" (UniqueName: \"kubernetes.io/projected/709d4f83-4488-4daf-855c-40523b4f14dd-kube-api-access-r4j47\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:56.068994 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068921    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/709d4f83-4488-4daf-855c-40523b4f14dd-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:56.068994 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "709d4f83-4488-4daf-855c-40523b4f14dd" (UID: "709d4f83-4488-4daf-855c-40523b4f14dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:56.068994 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.068986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "709d4f83-4488-4daf-855c-40523b4f14dd" (UID: "709d4f83-4488-4daf-855c-40523b4f14dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:56.115221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.115187    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:22:56.115380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.115275    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:22:56.169494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.169433    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:56.169494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.169469    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/709d4f83-4488-4daf-855c-40523b4f14dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:22:56.885442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.885409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerStarted","Data":"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"}
Sep 09 20:22:56.885934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.885481    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerStarted","Data":"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"}
Sep 09 20:22:56.885934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.885705    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:56.887475 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.887451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62" event={"ID":"709d4f83-4488-4daf-855c-40523b4f14dd","Type":"ContainerDied","Data":"bcc99da5f85a47b812e345672021d4b31f7b88d673b627e3c8affbd5c2825cb9"}
Sep 09 20:22:56.887602 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.887485    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"
Sep 09 20:22:56.887602 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.887503    2564 scope.go:117] "RemoveContainer" containerID="901b25c05d30081f3e278182b9f9cf8e88740fad76348d38d19c928ed6ecf9cf"
Sep 09 20:22:56.896522 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.896499    2564 scope.go:117] "RemoveContainer" containerID="06aa3ec9a2838a23a871dbe21d26f6d6de14c22f1a5ca16b7c92ea2eadde51a8"
Sep 09 20:22:56.904965 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.904908    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podStartSLOduration=3.079777373 podStartE2EDuration="6.904891114s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.825113741s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:56.902758109 +0000 UTC m=+716.277296204" watchObservedRunningTime="2026-09-09 20:22:56.904891114 +0000 UTC m=+716.279429256"
Sep 09 20:22:56.905425 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.905399    2564 scope.go:117] "RemoveContainer" containerID="2efbba342e1d2c708db9cb8331348c5402cbbdd7267a296cc016d4dfc928e096"
Sep 09 20:22:56.916487 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.916458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:22:56.920001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:56.919974    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5c8b67b6bd-hzq62"]
Sep 09 20:22:57.119097 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:57.119044    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" path="/var/lib/kubelet/pods/709d4f83-4488-4daf-855c-40523b4f14dd/volumes"
Sep 09 20:22:57.893399 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:57.893363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:22:57.894724 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:57.894696    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:22:58.898040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:22:58.897991    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:03.902872 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:03.902839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:23:03.903401 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:03.903328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:05.653479 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.653437    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:23:05.653877 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.653812    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" containerID="cri-o://08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd" gracePeriod=30
Sep 09 20:23:05.653877 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.653827    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" containerID="cri-o://333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3" gracePeriod=30
Sep 09 20:23:05.654041 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.653778    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" containerID="cri-o://c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1" gracePeriod=30
Sep 09 20:23:05.935979 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.935893    2564 generic.go:356] "Generic (PLEG): container finished" podID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerID="333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3" exitCode=2
Sep 09 20:23:05.935979 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:05.935940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerDied","Data":"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"}
Sep 09 20:23:06.113969 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:06.113925    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:23:06.114385 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:06.114352    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:10.434084 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:10.434014    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:10.963760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:10.963725    2564 generic.go:356] "Generic (PLEG): container finished" podID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerID="c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1" exitCode=0
Sep 09 20:23:10.963935 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:10.963771    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerDied","Data":"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"}
Sep 09 20:23:13.903413 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:13.903376    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:15.434473 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:15.434432    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:16.113874 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:16.113829    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:23:16.114190 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:16.114165    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:20.434166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:20.434115    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:20.434568 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:20.434280    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:23:23.903354 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:23.903308    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:25.433789 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:25.433745    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:26.114463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:26.114417    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:23:26.114679 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:26.114546    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:23:26.114771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:26.114744    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:26.114855 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:26.114843    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:23:30.433658 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:30.433609    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:33.903409 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:33.903368    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:35.434317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.434267    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 09 20:23:35.803201 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.803174    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:23:35.892354 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config\") pod \"974a1497-4814-47ee-92dc-ad0c1322ebca\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") "
Sep 09 20:23:35.892548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892377    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") pod \"974a1497-4814-47ee-92dc-ad0c1322ebca\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") "
Sep 09 20:23:35.892548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9m8s\" (UniqueName: \"kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s\") pod \"974a1497-4814-47ee-92dc-ad0c1322ebca\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") "
Sep 09 20:23:35.892548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892477    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert\") pod \"974a1497-4814-47ee-92dc-ad0c1322ebca\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") "
Sep 09 20:23:35.892548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892506    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location\") pod \"974a1497-4814-47ee-92dc-ad0c1322ebca\" (UID: \"974a1497-4814-47ee-92dc-ad0c1322ebca\") "
Sep 09 20:23:35.892809 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "974a1497-4814-47ee-92dc-ad0c1322ebca" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:35.892896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "974a1497-4814-47ee-92dc-ad0c1322ebca" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:35.892896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.892883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "974a1497-4814-47ee-92dc-ad0c1322ebca" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:23:35.894718 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.894689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s" (OuterVolumeSpecName: "kube-api-access-x9m8s") pod "974a1497-4814-47ee-92dc-ad0c1322ebca" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca"). InnerVolumeSpecName "kube-api-access-x9m8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:35.894845 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.894819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "974a1497-4814-47ee-92dc-ad0c1322ebca" (UID: "974a1497-4814-47ee-92dc-ad0c1322ebca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:35.993671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.993571    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:23:35.993671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.993605    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/974a1497-4814-47ee-92dc-ad0c1322ebca-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:23:35.993671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.993615    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9m8s\" (UniqueName: \"kubernetes.io/projected/974a1497-4814-47ee-92dc-ad0c1322ebca-kube-api-access-x9m8s\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:23:35.993671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.993626    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/974a1497-4814-47ee-92dc-ad0c1322ebca-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:23:35.993671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:35.993634    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/974a1497-4814-47ee-92dc-ad0c1322ebca-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:23:36.096281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.096242    2564 generic.go:356] "Generic (PLEG): container finished" podID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerID="08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd" exitCode=0
Sep 09 20:23:36.096450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.096289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerDied","Data":"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"}
Sep 09 20:23:36.096450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.096305    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"
Sep 09 20:23:36.096450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.096356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5" event={"ID":"974a1497-4814-47ee-92dc-ad0c1322ebca","Type":"ContainerDied","Data":"7d793707362f90ea6004afff1cadd44d1ab0b07475055b0048c484bd3d416427"}
Sep 09 20:23:36.096450 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.096390    2564 scope.go:117] "RemoveContainer" containerID="08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"
Sep 09 20:23:36.105625 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.105604    2564 scope.go:117] "RemoveContainer" containerID="333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"
Sep 09 20:23:36.113626 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.113604    2564 scope.go:117] "RemoveContainer" containerID="c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"
Sep 09 20:23:36.119081 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.119034    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:23:36.121548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.121525    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-43f1f-predictor-5b55655cfb-szbx5"]
Sep 09 20:23:36.123274 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.123258    2564 scope.go:117] "RemoveContainer" containerID="54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"
Sep 09 20:23:36.131096 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131070    2564 scope.go:117] "RemoveContainer" containerID="08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"
Sep 09 20:23:36.131409 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:23:36.131389    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd\": container with ID starting with 08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd not found: ID does not exist" containerID="08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"
Sep 09 20:23:36.131493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131422    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd"} err="failed to get container status \"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd\": rpc error: code = NotFound desc = could not find container \"08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd\": container with ID starting with 08f98991d0feda6a05ea112a499ddfdf040c7afd8b06acbb60eed223c2191fcd not found: ID does not exist"
Sep 09 20:23:36.131493 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131451    2564 scope.go:117] "RemoveContainer" containerID="333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"
Sep 09 20:23:36.131736 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:23:36.131717    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3\": container with ID starting with 333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3 not found: ID does not exist" containerID="333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"
Sep 09 20:23:36.131775 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131742    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3"} err="failed to get container status \"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3\": rpc error: code = NotFound desc = could not find container \"333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3\": container with ID starting with 333ec91f25368b4d41460ee0295fe90fce3a3983f3ba900d4b5002da4bb847f3 not found: ID does not exist"
Sep 09 20:23:36.131775 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131755    2564 scope.go:117] "RemoveContainer" containerID="c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"
Sep 09 20:23:36.131994 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:23:36.131977    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1\": container with ID starting with c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1 not found: ID does not exist" containerID="c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"
Sep 09 20:23:36.132042 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.131997    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1"} err="failed to get container status \"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1\": rpc error: code = NotFound desc = could not find container \"c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1\": container with ID starting with c4eb60046aa53f293a0f563c9d7f41aa3bd854376b7508a99ea7cc97270840e1 not found: ID does not exist"
Sep 09 20:23:36.132042 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.132007    2564 scope.go:117] "RemoveContainer" containerID="54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"
Sep 09 20:23:36.132296 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:23:36.132277    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb\": container with ID starting with 54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb not found: ID does not exist" containerID="54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"
Sep 09 20:23:36.132365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:36.132302    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb"} err="failed to get container status \"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb\": rpc error: code = NotFound desc = could not find container \"54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb\": container with ID starting with 54582346cd77c1c38774601527c985f5cb3adb1fe99ed85ee42b7ba769cd93eb not found: ID does not exist"
Sep 09 20:23:37.118753 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:37.118718    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" path="/var/lib/kubelet/pods/974a1497-4814-47ee-92dc-ad0c1322ebca/volumes"
Sep 09 20:23:43.903456 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:43.903408    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:23:53.904044 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:23:53.904002    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:24:03.903408 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:03.903361    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:24:13.904380 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:13.904350    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:24:20.701105 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:20.701052    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:24:20.701553 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:20.701380    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" containerID="cri-o://e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31" gracePeriod=30
Sep 09 20:24:20.701553 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:20.701450    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kube-rbac-proxy" containerID="cri-o://1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6" gracePeriod=30
Sep 09 20:24:21.328144 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:21.328106    2564 generic.go:356] "Generic (PLEG): container finished" podID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerID="1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6" exitCode=2
Sep 09 20:24:21.328296 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:21.328181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerDied","Data":"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"}
Sep 09 20:24:23.898570 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:23.898473    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:24:23.903848 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:23.903814    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:24:25.851338 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.851311    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:24:25.907530 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.907405    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:24:25.907980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.907959    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="storage-initializer"
Sep 09 20:24:25.907980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.907977    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="storage-initializer"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908000    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908008    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908016    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="storage-initializer"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908021    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="storage-initializer"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908036    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908042    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908050    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908117    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908134    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908142    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908155    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="storage-initializer"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908162    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="storage-initializer"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908171    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908187    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908194    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kube-rbac-proxy"
Sep 09 20:24:25.908188 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908201    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908206    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908214    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908219    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908225    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="storage-initializer"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908231    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="storage-initializer"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908237    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908242    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908304    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="agent"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908313    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908319    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kube-rbac-proxy"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908324    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908331    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="974a1497-4814-47ee-92dc-ad0c1322ebca" containerName="kube-rbac-proxy"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908336    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1dba65-9732-4d63-88cf-177ecd33c763" containerName="kserve-container"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908342    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerName="kube-rbac-proxy"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908348    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kube-rbac-proxy"
Sep 09 20:24:25.909085 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.908354    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="709d4f83-4488-4daf-855c-40523b4f14dd" containerName="kserve-container"
Sep 09 20:24:25.911666 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.911642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:25.913920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.913895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:24:25.914103 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.913938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 20:24:25.921204 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.921181    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:24:25.940002 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.939962    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location\") pod \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") "
Sep 09 20:24:25.940221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940094    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert\") pod \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") "
Sep 09 20:24:25.940221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940131    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls\") pod \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") "
Sep 09 20:24:25.940221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940165    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config\") pod \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") "
Sep 09 20:24:25.940221 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940207    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggngk\" (UniqueName: \"kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk\") pod \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\" (UID: \"51b019bf-ddeb-4378-a8d7-a12d493dc9fe\") "
Sep 09 20:24:25.940446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940360    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51b019bf-ddeb-4378-a8d7-a12d493dc9fe" (UID: "51b019bf-ddeb-4378-a8d7-a12d493dc9fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:25.940504 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940484    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51b019bf-ddeb-4378-a8d7-a12d493dc9fe" (UID: "51b019bf-ddeb-4378-a8d7-a12d493dc9fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:25.940640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51b019bf-ddeb-4378-a8d7-a12d493dc9fe" (UID: "51b019bf-ddeb-4378-a8d7-a12d493dc9fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:25.940640 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.940552    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.942906 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.942877    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51b019bf-ddeb-4378-a8d7-a12d493dc9fe" (UID: "51b019bf-ddeb-4378-a8d7-a12d493dc9fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:25.943249 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.943222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk" (OuterVolumeSpecName: "kube-api-access-ggngk") pod "51b019bf-ddeb-4378-a8d7-a12d493dc9fe" (UID: "51b019bf-ddeb-4378-a8d7-a12d493dc9fe"). InnerVolumeSpecName "kube-api-access-ggngk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:25.944112 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.944088    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:24:25.947870 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.947849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:25.950046 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.950024    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:24:25.957220 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:25.957194    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:24:26.041503 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.041675 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.041675 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.041675 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.041675 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.041822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs5v\" (UniqueName: \"kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.041822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84w7\" (UniqueName: \"kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.041822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.041822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.041822 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.042001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041849    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggngk\" (UniqueName: \"kubernetes.io/projected/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-api-access-ggngk\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:24:26.042001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041861    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:24:26.042001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041872    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:24:26.042001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.041886    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51b019bf-ddeb-4378-a8d7-a12d493dc9fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:24:26.143324 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.143324 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs5v\" (UniqueName: \"kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84w7\" (UniqueName: \"kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.143604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.144033 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.144033 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:26.143882    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 20:24:26.144033 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:26.143947    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls podName:307adf6e-a272-4e68-8064-7ebf677e3dd1 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:26.643927984 +0000 UTC m=+806.018466060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls") pod "isvc-canary-stop-predictor-69f8f5d9cc-b6phq" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 20:24:26.144033 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.143951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.144365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.144174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.144365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.144214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.144365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.144311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.144530 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.144491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.146018 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.145998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.152174 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.152149    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84w7\" (UniqueName: \"kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.152283 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.152186    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs5v\" (UniqueName: \"kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v\") pod \"isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.223982 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.223888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:26.355690 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.355655    2564 generic.go:356] "Generic (PLEG): container finished" podID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" containerID="e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31" exitCode=0
Sep 09 20:24:26.355868 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.355707    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"
Sep 09 20:24:26.355868 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.355753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerDied","Data":"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"}
Sep 09 20:24:26.355868 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.355817    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7" event={"ID":"51b019bf-ddeb-4378-a8d7-a12d493dc9fe","Type":"ContainerDied","Data":"99a0e10072500ae3cd3f4f9942de107ea0b00d35a870dd1715c123206dc1568a"}
Sep 09 20:24:26.355868 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.355851    2564 scope.go:117] "RemoveContainer" containerID="1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"
Sep 09 20:24:26.365441 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.365416    2564 scope.go:117] "RemoveContainer" containerID="e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"
Sep 09 20:24:26.373814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.373792    2564 scope.go:117] "RemoveContainer" containerID="4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"
Sep 09 20:24:26.380407 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.380189    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:24:26.382645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.382617    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-t6cq7"]
Sep 09 20:24:26.385514 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.385488    2564 scope.go:117] "RemoveContainer" containerID="1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"
Sep 09 20:24:26.385927 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:26.385907    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6\": container with ID starting with 1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6 not found: ID does not exist" containerID="1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"
Sep 09 20:24:26.386003 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.385937    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6"} err="failed to get container status \"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6\": rpc error: code = NotFound desc = could not find container \"1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6\": container with ID starting with 1810710630d03bf22f1d2d6b4d55bf704aa5040112557e0d416acba9755206e6 not found: ID does not exist"
Sep 09 20:24:26.386003 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.385960    2564 scope.go:117] "RemoveContainer" containerID="e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"
Sep 09 20:24:26.386307 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:26.386282    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31\": container with ID starting with e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31 not found: ID does not exist" containerID="e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"
Sep 09 20:24:26.386454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.386310    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31"} err="failed to get container status \"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31\": rpc error: code = NotFound desc = could not find container \"e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31\": container with ID starting with e40a73c7dc1291a65c49a887553e19b4c4cadd128438639e875a3ac37ded1d31 not found: ID does not exist"
Sep 09 20:24:26.386454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.386327    2564 scope.go:117] "RemoveContainer" containerID="4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"
Sep 09 20:24:26.386668 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:26.386642    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1\": container with ID starting with 4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1 not found: ID does not exist" containerID="4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"
Sep 09 20:24:26.386711 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.386680    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1"} err="failed to get container status \"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1\": rpc error: code = NotFound desc = could not find container \"4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1\": container with ID starting with 4b154c6c9311e0977d77a3521f547dc22bbabe863410d1ad59e88b92feddaca1 not found: ID does not exist"
Sep 09 20:24:26.507681 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.507459    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:24:26.510568 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:24:26.510540    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9e345b_f9c0_4ff0_801c_f263f15d7763.slice/crio-4aab5bbad432062275b0436a307a8e619c80e83d6bdedf701e6819b6ba3e8d11 WatchSource:0}: Error finding container 4aab5bbad432062275b0436a307a8e619c80e83d6bdedf701e6819b6ba3e8d11: Status 404 returned error can't find the container with id 4aab5bbad432062275b0436a307a8e619c80e83d6bdedf701e6819b6ba3e8d11
Sep 09 20:24:26.647169 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.647141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.649558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.649528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-b6phq\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:26.867411 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:26.867369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:27.119898 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.119817    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b019bf-ddeb-4378-a8d7-a12d493dc9fe" path="/var/lib/kubelet/pods/51b019bf-ddeb-4378-a8d7-a12d493dc9fe/volumes"
Sep 09 20:24:27.168738 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.168706    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:24:27.170883 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:24:27.170858    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307adf6e_a272_4e68_8064_7ebf677e3dd1.slice/crio-23b64d25d12122551d77bd88061120d53f4f48d280c3f315495e6b5ea918966d WatchSource:0}: Error finding container 23b64d25d12122551d77bd88061120d53f4f48d280c3f315495e6b5ea918966d: Status 404 returned error can't find the container with id 23b64d25d12122551d77bd88061120d53f4f48d280c3f315495e6b5ea918966d
Sep 09 20:24:27.361946 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.361906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerStarted","Data":"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"}
Sep 09 20:24:27.362181 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.361990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerStarted","Data":"4aab5bbad432062275b0436a307a8e619c80e83d6bdedf701e6819b6ba3e8d11"}
Sep 09 20:24:27.364521 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.364491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerStarted","Data":"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"}
Sep 09 20:24:27.364634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:27.364545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerStarted","Data":"23b64d25d12122551d77bd88061120d53f4f48d280c3f315495e6b5ea918966d"}
Sep 09 20:24:31.384443 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:31.384403    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerID="e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c" exitCode=0
Sep 09 20:24:31.384868 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:31.384476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerDied","Data":"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"}
Sep 09 20:24:31.386252 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:31.386229    2564 generic.go:356] "Generic (PLEG): container finished" podID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerID="ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94" exitCode=0
Sep 09 20:24:31.386352 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:31.386300    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerDied","Data":"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"}
Sep 09 20:24:32.394121 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.394085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerStarted","Data":"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"}
Sep 09 20:24:32.394631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.394158    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerStarted","Data":"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"}
Sep 09 20:24:32.394631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.394367    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:32.396207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.396181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerStarted","Data":"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"}
Sep 09 20:24:32.396339 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.396238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerStarted","Data":"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"}
Sep 09 20:24:32.396456 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.396436    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:32.413759 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.413707    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podStartSLOduration=3.544400666 podStartE2EDuration="7.413691113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869290447s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:32.411782909 +0000 UTC m=+811.786321003" watchObservedRunningTime="2026-09-09 20:24:32.413691113 +0000 UTC m=+811.788229206"
Sep 09 20:24:32.432357 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:32.432277    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podStartSLOduration=3.5229259859999997 podStartE2EDuration="7.432254789s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.909328803s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:32.428850207 +0000 UTC m=+811.803388300" watchObservedRunningTime="2026-09-09 20:24:32.432254789 +0000 UTC m=+811.806792884"
Sep 09 20:24:33.402763 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:33.402728    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:33.403183 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:33.402791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:33.403777 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:33.403750    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:24:33.403871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:33.403816    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:24:34.407309 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:34.407266    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:24:34.407702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:34.407315    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:24:39.415133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:39.413101    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:24:39.415133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:39.414708    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:24:39.415670 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:39.415542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:24:39.416476 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:39.416444    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:24:49.413721 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:49.413676    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:24:49.417006 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:49.416970    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:24:56.118981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.118943    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:24:56.123416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.123390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.125641 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.125619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2aad0-predictor-serving-cert\""
Sep 09 20:24:56.125768 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.125628    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-2aad0-kube-rbac-proxy-sar-config\""
Sep 09 20:24:56.133355 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.133329    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:24:56.205805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.205759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjxs\" (UniqueName: \"kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.206004 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.205821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.206004 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.205858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.206004 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.205883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.206004 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.205910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.306706 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.306670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.306706 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.306711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.306980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.306731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.306980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.306815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjxs\" (UniqueName: \"kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.306980 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:56.306856    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-2aad0-predictor-serving-cert" not found
Sep 09 20:24:56.306980 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:24:56.306937    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls podName:7a7ae7c2-5021-4617-9cda-9b1f0cd16473 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:56.806914062 +0000 UTC m=+836.181452151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473") : secret "isvc-sklearn-graph-raw-hpa-2aad0-predictor-serving-cert" not found
Sep 09 20:24:56.306980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.306962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.307367 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.307218    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.307637 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.307615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.307637 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.307626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.314906 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.314883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjxs\" (UniqueName: \"kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.809996 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.809952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:56.812496 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:56.812474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:57.035240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:57.035186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:24:57.318877 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:57.318836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:24:57.322415 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:24:57.322388    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ae7c2_5021_4617_9cda_9b1f0cd16473.slice/crio-b61d807eaf02f2380ac32ebcc427b49c0eb22d4bb7c2c8bace317cffe47c8e4c WatchSource:0}: Error finding container b61d807eaf02f2380ac32ebcc427b49c0eb22d4bb7c2c8bace317cffe47c8e4c: Status 404 returned error can't find the container with id b61d807eaf02f2380ac32ebcc427b49c0eb22d4bb7c2c8bace317cffe47c8e4c
Sep 09 20:24:57.522573 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:57.522536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerStarted","Data":"298f2c07eb6a23d36ae59c45433dfaaaec988f437fa55e942b758f4929aeaf79"}
Sep 09 20:24:57.522748 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:57.522597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerStarted","Data":"b61d807eaf02f2380ac32ebcc427b49c0eb22d4bb7c2c8bace317cffe47c8e4c"}
Sep 09 20:24:59.413880 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:59.413842    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:24:59.416089 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:24:59.416045    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:01.546030 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:01.545990    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerID="298f2c07eb6a23d36ae59c45433dfaaaec988f437fa55e942b758f4929aeaf79" exitCode=0
Sep 09 20:25:01.546443 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:01.546044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerDied","Data":"298f2c07eb6a23d36ae59c45433dfaaaec988f437fa55e942b758f4929aeaf79"}
Sep 09 20:25:02.554261 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.554224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerStarted","Data":"ddcc7390f45eb5fbc351c40f04b9d68d618b4cd813761ad2403a9d841777923a"}
Sep 09 20:25:02.554735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.554300    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerStarted","Data":"3089a8dfc65cf7a0df6386ac45e323abc9fb8d41f7a9efdd2e9f1fe8711af92a"}
Sep 09 20:25:02.554735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.554576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:25:02.554735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.554626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:25:02.556167 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.556138    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:02.573508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:02.573452    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podStartSLOduration=2.903865167 podStartE2EDuration="6.573431942s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.669566775s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:02.571765099 +0000 UTC m=+841.946303186" watchObservedRunningTime="2026-09-09 20:25:02.573431942 +0000 UTC m=+841.947970035"
Sep 09 20:25:03.559143 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:03.559100    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:08.563803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:08.563767    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:25:08.564291 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:08.564262    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:09.414224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:09.414186    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:25:09.416485 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:09.416461    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:18.564266 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:18.564227    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:19.413763 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:19.413721    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:25:19.417002 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:19.416979    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:28.564774 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:28.564735    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:29.413825 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:29.413784    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:25:29.417081 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:29.417039    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:38.564563 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:38.564527    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:39.413818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:39.413781    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:25:39.417100 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:39.417049    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:48.565194 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:48.565147    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:49.119463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.119419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:25:49.229182 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.229132    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:25:49.229665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.229434    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container" containerID="cri-o://ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d" gracePeriod=30
Sep 09 20:25:49.229665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.229460    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kube-rbac-proxy" containerID="cri-o://178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765" gracePeriod=30
Sep 09 20:25:49.230527 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.230298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:25:49.407554 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.407458    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 20:25:49.792506 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.792470    2564 generic.go:356] "Generic (PLEG): container finished" podID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerID="178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765" exitCode=2
Sep 09 20:25:49.792886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:49.792539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerDied","Data":"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"}
Sep 09 20:25:54.387252 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.387221    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:25:54.444972 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.444874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert\") pod \"307adf6e-a272-4e68-8064-7ebf677e3dd1\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") "
Sep 09 20:25:54.444972 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.444920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config\") pod \"307adf6e-a272-4e68-8064-7ebf677e3dd1\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") "
Sep 09 20:25:54.445242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location\") pod \"307adf6e-a272-4e68-8064-7ebf677e3dd1\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") "
Sep 09 20:25:54.445242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") pod \"307adf6e-a272-4e68-8064-7ebf677e3dd1\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") "
Sep 09 20:25:54.445242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445236    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r84w7\" (UniqueName: \"kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7\") pod \"307adf6e-a272-4e68-8064-7ebf677e3dd1\" (UID: \"307adf6e-a272-4e68-8064-7ebf677e3dd1\") "
Sep 09 20:25:54.445386 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445304    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "307adf6e-a272-4e68-8064-7ebf677e3dd1" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:54.445386 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445313    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "307adf6e-a272-4e68-8064-7ebf677e3dd1" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:54.445386 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445304    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "307adf6e-a272-4e68-8064-7ebf677e3dd1" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:54.445595 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445570    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:25:54.445595 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445598    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:25:54.445783 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.445614    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307adf6e-a272-4e68-8064-7ebf677e3dd1-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:25:54.447327 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.447299    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7" (OuterVolumeSpecName: "kube-api-access-r84w7") pod "307adf6e-a272-4e68-8064-7ebf677e3dd1" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1"). InnerVolumeSpecName "kube-api-access-r84w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:54.447427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.447348    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "307adf6e-a272-4e68-8064-7ebf677e3dd1" (UID: "307adf6e-a272-4e68-8064-7ebf677e3dd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:54.546248 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.546196    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307adf6e-a272-4e68-8064-7ebf677e3dd1-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:25:54.546248 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.546242    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84w7\" (UniqueName: \"kubernetes.io/projected/307adf6e-a272-4e68-8064-7ebf677e3dd1-kube-api-access-r84w7\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:25:54.823885 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.823850    2564 generic.go:356] "Generic (PLEG): container finished" podID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerID="ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d" exitCode=0
Sep 09 20:25:54.824116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.823899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerDied","Data":"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"}
Sep 09 20:25:54.824116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.823903    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"
Sep 09 20:25:54.824116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.823988    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq" event={"ID":"307adf6e-a272-4e68-8064-7ebf677e3dd1","Type":"ContainerDied","Data":"23b64d25d12122551d77bd88061120d53f4f48d280c3f315495e6b5ea918966d"}
Sep 09 20:25:54.824116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.824029    2564 scope.go:117] "RemoveContainer" containerID="178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"
Sep 09 20:25:54.833453 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.833436    2564 scope.go:117] "RemoveContainer" containerID="ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"
Sep 09 20:25:54.842465 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.842438    2564 scope.go:117] "RemoveContainer" containerID="ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"
Sep 09 20:25:54.845272 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.845244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:25:54.848864 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.848841    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-b6phq"]
Sep 09 20:25:54.852138 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.852117    2564 scope.go:117] "RemoveContainer" containerID="178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"
Sep 09 20:25:54.852478 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:25:54.852458    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765\": container with ID starting with 178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765 not found: ID does not exist" containerID="178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"
Sep 09 20:25:54.852536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.852488    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765"} err="failed to get container status \"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765\": rpc error: code = NotFound desc = could not find container \"178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765\": container with ID starting with 178bcf486e688d7cd2c99d70dbdf673ed1f378e94c3d394de269ba354914d765 not found: ID does not exist"
Sep 09 20:25:54.852536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.852506    2564 scope.go:117] "RemoveContainer" containerID="ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"
Sep 09 20:25:54.852812 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:25:54.852797    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d\": container with ID starting with ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d not found: ID does not exist" containerID="ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"
Sep 09 20:25:54.852853 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.852815    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d"} err="failed to get container status \"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d\": rpc error: code = NotFound desc = could not find container \"ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d\": container with ID starting with ce0d91dafbfce9364f52c5af33e9e9ad930182376925291aafdc20b2c721583d not found: ID does not exist"
Sep 09 20:25:54.852853 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.852829    2564 scope.go:117] "RemoveContainer" containerID="ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"
Sep 09 20:25:54.853044 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:25:54.853028    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94\": container with ID starting with ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94 not found: ID does not exist" containerID="ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"
Sep 09 20:25:54.853159 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:54.853048    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94"} err="failed to get container status \"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94\": rpc error: code = NotFound desc = could not find container \"ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94\": container with ID starting with ee674a34b0361b054a1eeff2b14ff0753679d75d066f70d00c8e202041ef2b94 not found: ID does not exist"
Sep 09 20:25:55.119457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:55.119365    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" path="/var/lib/kubelet/pods/307adf6e-a272-4e68-8064-7ebf677e3dd1/volumes"
Sep 09 20:25:55.931522 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:55.931484    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:25:55.932008 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:55.931721    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" containerID="cri-o://5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39" gracePeriod=30
Sep 09 20:25:55.932008 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:55.931775    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kube-rbac-proxy" containerID="cri-o://7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd" gracePeriod=30
Sep 09 20:25:56.837815 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:56.837782    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerID="7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd" exitCode=2
Sep 09 20:25:56.837991 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:56.837870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerDied","Data":"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"}
Sep 09 20:25:58.565119 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:58.565080    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:25:59.114974 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:59.114927    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:25:59.408410 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:25:59.408303    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 09 20:26:01.081478 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.081451    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:26:01.082857 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.082833    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:26:01.088546 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.088525    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:26:01.089943 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.089913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:26:01.192304 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.192232    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:26:01.306050 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306017    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config\") pod \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") "
Sep 09 20:26:01.306050 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306076    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls\") pod \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") "
Sep 09 20:26:01.306389 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location\") pod \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") "
Sep 09 20:26:01.306389 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert\") pod \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") "
Sep 09 20:26:01.306389 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzs5v\" (UniqueName: \"kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v\") pod \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\" (UID: \"4e9e345b-f9c0-4ff0-801c-f263f15d7763\") "
Sep 09 20:26:01.306550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306454    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e9e345b-f9c0-4ff0-801c-f263f15d7763" (UID: "4e9e345b-f9c0-4ff0-801c-f263f15d7763"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:01.306550 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e9e345b-f9c0-4ff0-801c-f263f15d7763" (UID: "4e9e345b-f9c0-4ff0-801c-f263f15d7763"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:01.306653 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.306541    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e9e345b-f9c0-4ff0-801c-f263f15d7763" (UID: "4e9e345b-f9c0-4ff0-801c-f263f15d7763"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:01.308336 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.308308    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e9e345b-f9c0-4ff0-801c-f263f15d7763" (UID: "4e9e345b-f9c0-4ff0-801c-f263f15d7763"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:01.308426 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.308363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v" (OuterVolumeSpecName: "kube-api-access-pzs5v") pod "4e9e345b-f9c0-4ff0-801c-f263f15d7763" (UID: "4e9e345b-f9c0-4ff0-801c-f263f15d7763"). InnerVolumeSpecName "kube-api-access-pzs5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:01.407483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.407427    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzs5v\" (UniqueName: \"kubernetes.io/projected/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-api-access-pzs5v\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.407483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.407478    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.407483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.407490    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e9e345b-f9c0-4ff0-801c-f263f15d7763-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.407483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.407500    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e9e345b-f9c0-4ff0-801c-f263f15d7763-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.407771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.407508    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e9e345b-f9c0-4ff0-801c-f263f15d7763-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.867617 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.867588    2564 generic.go:356] "Generic (PLEG): container finished" podID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerID="5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39" exitCode=0
Sep 09 20:26:01.867798 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.867650    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"
Sep 09 20:26:01.867798 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.867672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerDied","Data":"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"}
Sep 09 20:26:01.867798 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.867746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p" event={"ID":"4e9e345b-f9c0-4ff0-801c-f263f15d7763","Type":"ContainerDied","Data":"4aab5bbad432062275b0436a307a8e619c80e83d6bdedf701e6819b6ba3e8d11"}
Sep 09 20:26:01.867798 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.867786    2564 scope.go:117] "RemoveContainer" containerID="7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"
Sep 09 20:26:01.884593 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.884573    2564 scope.go:117] "RemoveContainer" containerID="5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"
Sep 09 20:26:01.893125 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.893091    2564 scope.go:117] "RemoveContainer" containerID="e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"
Sep 09 20:26:01.894635 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.894612    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:26:01.897610 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.897587    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-7f9ff7fcd4-rsg4p"]
Sep 09 20:26:01.902110 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.902085    2564 scope.go:117] "RemoveContainer" containerID="7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"
Sep 09 20:26:01.902460 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:26:01.902443    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd\": container with ID starting with 7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd not found: ID does not exist" containerID="7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"
Sep 09 20:26:01.902504 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.902471    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd"} err="failed to get container status \"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd\": rpc error: code = NotFound desc = could not find container \"7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd\": container with ID starting with 7010aa6a462e78a76c6380016688671d083d66074d55c7f9c6cbebccea3e3ecd not found: ID does not exist"
Sep 09 20:26:01.902504 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.902489    2564 scope.go:117] "RemoveContainer" containerID="5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"
Sep 09 20:26:01.902768 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:26:01.902746    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39\": container with ID starting with 5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39 not found: ID does not exist" containerID="5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"
Sep 09 20:26:01.902805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.902776    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39"} err="failed to get container status \"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39\": rpc error: code = NotFound desc = could not find container \"5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39\": container with ID starting with 5e8091ecd3980b58e714b8437142e98abf11880d8b01fe6ef46afad3de340b39 not found: ID does not exist"
Sep 09 20:26:01.902805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.902793    2564 scope.go:117] "RemoveContainer" containerID="e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"
Sep 09 20:26:01.903035 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:26:01.903020    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c\": container with ID starting with e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c not found: ID does not exist" containerID="e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"
Sep 09 20:26:01.903081 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:01.903040    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c"} err="failed to get container status \"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c\": rpc error: code = NotFound desc = could not find container \"e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c\": container with ID starting with e5afc55e6717ec698c28fd66f38a53f242acc7333bfb49c0c9c3da97191c9d0c not found: ID does not exist"
Sep 09 20:26:03.119347 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:03.119313    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" path="/var/lib/kubelet/pods/4e9e345b-f9c0-4ff0-801c-f263f15d7763/volumes"
Sep 09 20:26:08.564384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:08.564344    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:26:18.564880 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:18.564849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:26:46.243713 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.243678    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:26:46.246378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.243930    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" containerID="cri-o://3089a8dfc65cf7a0df6386ac45e323abc9fb8d41f7a9efdd2e9f1fe8711af92a" gracePeriod=30
Sep 09 20:26:46.246378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.243971    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kube-rbac-proxy" containerID="cri-o://ddcc7390f45eb5fbc351c40f04b9d68d618b4cd813761ad2403a9d841777923a" gracePeriod=30
Sep 09 20:26:46.378410 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378377    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:26:46.378744 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378732    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="storage-initializer"
Sep 09 20:26:46.378792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378745    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="storage-initializer"
Sep 09 20:26:46.378792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378757    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="storage-initializer"
Sep 09 20:26:46.378792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378762    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="storage-initializer"
Sep 09 20:26:46.378792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378770    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kube-rbac-proxy"
Sep 09 20:26:46.378792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378776    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kube-rbac-proxy"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378794    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378800    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378806    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378811    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378818    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kube-rbac-proxy"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378823    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kube-rbac-proxy"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378900    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kube-rbac-proxy"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378906    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="307adf6e-a272-4e68-8064-7ebf677e3dd1" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378913    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kserve-container"
Sep 09 20:26:46.378951 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.378919    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9e345b-f9c0-4ff0-801c-f263f15d7763" containerName="kube-rbac-proxy"
Sep 09 20:26:46.381950 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.381931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.383860 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.383838    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-6734e-predictor-serving-cert\""
Sep 09 20:26:46.383964 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.383942    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-6734e-kube-rbac-proxy-sar-config\""
Sep 09 20:26:46.391843 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.391818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:26:46.473850 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.473808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.474039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.473873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.474039 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.473930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9l8\" (UniqueName: \"kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.574765 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.574726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.574971 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.574787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.574971 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.574813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9l8\" (UniqueName: \"kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.574971 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:26:46.574949    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-serving-cert: secret "message-dumper-raw-6734e-predictor-serving-cert" not found
Sep 09 20:26:46.575194 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:26:46.575018    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls podName:74395e5f-aca7-4702-b491-38ffd24d6ef2 nodeName:}" failed. No retries permitted until 2026-09-09 20:26:47.075003781 +0000 UTC m=+946.449541878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls") pod "message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" (UID: "74395e5f-aca7-4702-b491-38ffd24d6ef2") : secret "message-dumper-raw-6734e-predictor-serving-cert" not found
Sep 09 20:26:46.575524 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.575501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:46.583382 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:46.583355    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9l8\" (UniqueName: \"kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:47.079233 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.079193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:47.081704 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.081672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") pod \"message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:47.098740 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.098710    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerID="ddcc7390f45eb5fbc351c40f04b9d68d618b4cd813761ad2403a9d841777923a" exitCode=2
Sep 09 20:26:47.098885 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.098781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerDied","Data":"ddcc7390f45eb5fbc351c40f04b9d68d618b4cd813761ad2403a9d841777923a"}
Sep 09 20:26:47.293129 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.293099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:47.563933 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.563906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:26:47.566224 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:26:47.566201    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74395e5f_aca7_4702_b491_38ffd24d6ef2.slice/crio-eea250b51cf029d137df17362f553d1447da39ba3f63de6ce31dc3f6bae1ba34 WatchSource:0}: Error finding container eea250b51cf029d137df17362f553d1447da39ba3f63de6ce31dc3f6bae1ba34: Status 404 returned error can't find the container with id eea250b51cf029d137df17362f553d1447da39ba3f63de6ce31dc3f6bae1ba34
Sep 09 20:26:47.568544 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:47.568527    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:26:48.105089 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:48.105030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerStarted","Data":"eea250b51cf029d137df17362f553d1447da39ba3f63de6ce31dc3f6bae1ba34"}
Sep 09 20:26:48.559574 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:48.559533    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 20:26:48.564480 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:48.564446    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:26:49.112169 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.112073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerStarted","Data":"52ef81a5b1eb872d268023b0be730dde0de44bd3e86bea947b19a7f19fecd644"}
Sep 09 20:26:49.112323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.112172    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:49.112323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.112198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerStarted","Data":"53325b8680158c19660a9a83e0c85d393f18090d8b2d894f5b0e936b29d580a7"}
Sep 09 20:26:49.112323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.112219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:49.119671 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.119641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:26:49.128321 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:49.128145    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" podStartSLOduration=1.929988159 podStartE2EDuration="3.128125847s" totalImagesPullingTime="1.198137688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:49.127500722 +0000 UTC m=+948.502038817" watchObservedRunningTime="2026-09-09 20:26:49.128125847 +0000 UTC m=+948.502663941"
Sep 09 20:26:51.123735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.123704    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerID="3089a8dfc65cf7a0df6386ac45e323abc9fb8d41f7a9efdd2e9f1fe8711af92a" exitCode=0
Sep 09 20:26:51.124168 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.123773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerDied","Data":"3089a8dfc65cf7a0df6386ac45e323abc9fb8d41f7a9efdd2e9f1fe8711af92a"}
Sep 09 20:26:51.208684 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.208660    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:26:51.314587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314501    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config\") pod \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") "
Sep 09 20:26:51.314587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314536    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location\") pod \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") "
Sep 09 20:26:51.314587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrjxs\" (UniqueName: \"kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs\") pod \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") "
Sep 09 20:26:51.314891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") pod \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") "
Sep 09 20:26:51.314891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314673    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert\") pod \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\" (UID: \"7a7ae7c2-5021-4617-9cda-9b1f0cd16473\") "
Sep 09 20:26:51.315004 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314883    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a7ae7c2-5021-4617-9cda-9b1f0cd16473" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:51.315051 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.314906    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a7ae7c2-5021-4617-9cda-9b1f0cd16473" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:51.315135 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.315092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a7ae7c2-5021-4617-9cda-9b1f0cd16473" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:51.316771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.316748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a7ae7c2-5021-4617-9cda-9b1f0cd16473" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:51.316873 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.316809    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs" (OuterVolumeSpecName: "kube-api-access-jrjxs") pod "7a7ae7c2-5021-4617-9cda-9b1f0cd16473" (UID: "7a7ae7c2-5021-4617-9cda-9b1f0cd16473"). InnerVolumeSpecName "kube-api-access-jrjxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:51.415959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.415925    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:51.415959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.415953    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:51.415959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.415963    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:51.416219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.415973    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:51.416219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:51.415982    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjxs\" (UniqueName: \"kubernetes.io/projected/7a7ae7c2-5021-4617-9cda-9b1f0cd16473-kube-api-access-jrjxs\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:26:52.129996 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.129956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r" event={"ID":"7a7ae7c2-5021-4617-9cda-9b1f0cd16473","Type":"ContainerDied","Data":"b61d807eaf02f2380ac32ebcc427b49c0eb22d4bb7c2c8bace317cffe47c8e4c"}
Sep 09 20:26:52.130427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.130009    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"
Sep 09 20:26:52.130427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.130039    2564 scope.go:117] "RemoveContainer" containerID="ddcc7390f45eb5fbc351c40f04b9d68d618b4cd813761ad2403a9d841777923a"
Sep 09 20:26:52.139869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.139852    2564 scope.go:117] "RemoveContainer" containerID="3089a8dfc65cf7a0df6386ac45e323abc9fb8d41f7a9efdd2e9f1fe8711af92a"
Sep 09 20:26:52.148227 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.148201    2564 scope.go:117] "RemoveContainer" containerID="298f2c07eb6a23d36ae59c45433dfaaaec988f437fa55e942b758f4929aeaf79"
Sep 09 20:26:52.152191 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.152162    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:26:52.154563 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:52.154541    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-2aad0-predictor-95c8d8c44-9d25r"]
Sep 09 20:26:53.119560 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:53.119526    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" path="/var/lib/kubelet/pods/7a7ae7c2-5021-4617-9cda-9b1f0cd16473/volumes"
Sep 09 20:26:56.128213 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:26:56.128182    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:27:06.379309 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379271    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:27:06.379702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379678    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container"
Sep 09 20:27:06.379702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379690    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container"
Sep 09 20:27:06.379702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379702    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kube-rbac-proxy"
Sep 09 20:27:06.379800 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379707    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kube-rbac-proxy"
Sep 09 20:27:06.379800 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379720    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="storage-initializer"
Sep 09 20:27:06.379800 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379725    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="storage-initializer"
Sep 09 20:27:06.379800 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379779    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kserve-container"
Sep 09 20:27:06.379800 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.379787    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7ae7c2-5021-4617-9cda-9b1f0cd16473" containerName="kube-rbac-proxy"
Sep 09 20:27:06.384881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.384856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.387016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.386991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-6734e-kube-rbac-proxy-sar-config\""
Sep 09 20:27:06.387170 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.387034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:27:06.387289 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.387272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-6734e-predictor-serving-cert\""
Sep 09 20:27:06.387342 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.387275    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:27:06.393702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.393667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:27:06.445849 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.445814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.446025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.445872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xrv\" (UniqueName: \"kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.446025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.445949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.446025 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.446006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.446169 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.446042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.546771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.546737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.546942 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.546775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.546942 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.546829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.546942 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.546865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xrv\" (UniqueName: \"kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.547136 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.547004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.547603 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.547575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.547716 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.547687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.547815 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.547784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.549447 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.549427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.554764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.554736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xrv\" (UniqueName: \"kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv\") pod \"isvc-logger-raw-6734e-predictor-67f97574b6-pq49p\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") " pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.696697 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.696593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:06.972818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:06.972790    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:27:06.974888 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:27:06.974861    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3232b0_c14f_41bb_9ba4_f2fddda37939.slice/crio-bef5591c8f12355e8751cbe0f6ae404ee09f9cde5d10c3b2f49390440cf7fed6 WatchSource:0}: Error finding container bef5591c8f12355e8751cbe0f6ae404ee09f9cde5d10c3b2f49390440cf7fed6: Status 404 returned error can't find the container with id bef5591c8f12355e8751cbe0f6ae404ee09f9cde5d10c3b2f49390440cf7fed6
Sep 09 20:27:07.211329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:07.211290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerStarted","Data":"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"}
Sep 09 20:27:07.211486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:07.211371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerStarted","Data":"bef5591c8f12355e8751cbe0f6ae404ee09f9cde5d10c3b2f49390440cf7fed6"}
Sep 09 20:27:11.232494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:11.232461    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerID="dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4" exitCode=0
Sep 09 20:27:11.232985 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:11.232541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerDied","Data":"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"}
Sep 09 20:27:12.240667 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.240626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerStarted","Data":"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"}
Sep 09 20:27:12.241052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.240732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerStarted","Data":"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"}
Sep 09 20:27:12.241052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.240761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerStarted","Data":"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"}
Sep 09 20:27:12.241052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.241040    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:12.241248 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.241220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:12.242472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.242448    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:12.261265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:12.261209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podStartSLOduration=2.189138457 podStartE2EDuration="6.261191978s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.072053521s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:12.258472427 +0000 UTC m=+971.633010539" watchObservedRunningTime="2026-09-09 20:27:12.261191978 +0000 UTC m=+971.635730072"
Sep 09 20:27:13.246269 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:13.246227    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:13.246732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:13.246412    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:13.247611 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:13.247585    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:14.251135 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:14.251096    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:14.251635 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:14.251608    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:19.255488 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:19.255455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:27:19.256106 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:19.256045    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:19.256384 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:19.256352    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:29.256770 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:29.256668    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:29.257334 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:29.257101    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:39.256231 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:39.256191    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:39.256734 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:39.256707    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:49.256870 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:49.256819    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:49.257345 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:49.257319    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:59.256239 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:59.256198    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:27:59.256715 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:27:59.256684    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:09.256016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:09.255974    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:28:09.256592 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:09.256409    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:19.256886 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:19.256843    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:28:19.257331 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:19.257285    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:28.114265 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:28.114229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:28:28.114754 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:28.114318    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:28:41.387862 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.387831    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh_74395e5f-aca7-4702-b491-38ffd24d6ef2/kserve-container/0.log"
Sep 09 20:28:41.541875 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.541837    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:28:41.542239 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.542203    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" containerID="cri-o://47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12" gracePeriod=30
Sep 09 20:28:41.542508 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.542219    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" containerID="cri-o://f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152" gracePeriod=30
Sep 09 20:28:41.542627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.542279    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" containerID="cri-o://634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392" gracePeriod=30
Sep 09 20:28:41.573126 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.573094    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:28:41.573390 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.573358    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kserve-container" containerID="cri-o://53325b8680158c19660a9a83e0c85d393f18090d8b2d894f5b0e936b29d580a7" gracePeriod=30
Sep 09 20:28:41.573531 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.573409    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kube-rbac-proxy" containerID="cri-o://52ef81a5b1eb872d268023b0be730dde0de44bd3e86bea947b19a7f19fecd644" gracePeriod=30
Sep 09 20:28:41.690551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.690515    2564 generic.go:356] "Generic (PLEG): container finished" podID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerID="52ef81a5b1eb872d268023b0be730dde0de44bd3e86bea947b19a7f19fecd644" exitCode=2
Sep 09 20:28:41.690551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.690546    2564 generic.go:356] "Generic (PLEG): container finished" podID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerID="53325b8680158c19660a9a83e0c85d393f18090d8b2d894f5b0e936b29d580a7" exitCode=2
Sep 09 20:28:41.690772 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.690557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerDied","Data":"52ef81a5b1eb872d268023b0be730dde0de44bd3e86bea947b19a7f19fecd644"}
Sep 09 20:28:41.690772 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.690641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerDied","Data":"53325b8680158c19660a9a83e0c85d393f18090d8b2d894f5b0e936b29d580a7"}
Sep 09 20:28:41.693942 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.693912    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerID="634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392" exitCode=2
Sep 09 20:28:41.694119 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.694007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerDied","Data":"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"}
Sep 09 20:28:41.827272 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.827248    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:28:41.881710 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.881673    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config\") pod \"74395e5f-aca7-4702-b491-38ffd24d6ef2\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") "
Sep 09 20:28:41.881879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.881728    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") pod \"74395e5f-aca7-4702-b491-38ffd24d6ef2\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") "
Sep 09 20:28:41.881879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.881745    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm9l8\" (UniqueName: \"kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8\") pod \"74395e5f-aca7-4702-b491-38ffd24d6ef2\" (UID: \"74395e5f-aca7-4702-b491-38ffd24d6ef2\") "
Sep 09 20:28:41.882148 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.882122    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74395e5f-aca7-4702-b491-38ffd24d6ef2" (UID: "74395e5f-aca7-4702-b491-38ffd24d6ef2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:41.883946 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.883925    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8" (OuterVolumeSpecName: "kube-api-access-fm9l8") pod "74395e5f-aca7-4702-b491-38ffd24d6ef2" (UID: "74395e5f-aca7-4702-b491-38ffd24d6ef2"). InnerVolumeSpecName "kube-api-access-fm9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:41.884005 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.883926    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74395e5f-aca7-4702-b491-38ffd24d6ef2" (UID: "74395e5f-aca7-4702-b491-38ffd24d6ef2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:41.982891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.982794    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:28:41.982891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.982837    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74395e5f-aca7-4702-b491-38ffd24d6ef2-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:28:41.982891 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:41.982848    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9l8\" (UniqueName: \"kubernetes.io/projected/74395e5f-aca7-4702-b491-38ffd24d6ef2-kube-api-access-fm9l8\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:28:42.706215 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.706171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh" event={"ID":"74395e5f-aca7-4702-b491-38ffd24d6ef2","Type":"ContainerDied","Data":"eea250b51cf029d137df17362f553d1447da39ba3f63de6ce31dc3f6bae1ba34"}
Sep 09 20:28:42.706680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.706239    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"
Sep 09 20:28:42.706680 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.706284    2564 scope.go:117] "RemoveContainer" containerID="52ef81a5b1eb872d268023b0be730dde0de44bd3e86bea947b19a7f19fecd644"
Sep 09 20:28:42.715297 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.715279    2564 scope.go:117] "RemoveContainer" containerID="53325b8680158c19660a9a83e0c85d393f18090d8b2d894f5b0e936b29d580a7"
Sep 09 20:28:42.728762 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.728732    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:28:42.732732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:42.732702    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-6734e-predictor-6fbfc74675-bz4wh"]
Sep 09 20:28:43.119764 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:43.119730    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" path="/var/lib/kubelet/pods/74395e5f-aca7-4702-b491-38ffd24d6ef2/volumes"
Sep 09 20:28:44.251897 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:44.251848    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:28:46.728387 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:46.728292    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerID="47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12" exitCode=0
Sep 09 20:28:46.728743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:46.728374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerDied","Data":"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"}
Sep 09 20:28:48.113792 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:48.113750    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:28:48.114328 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:48.114115    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:49.251260 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:49.251215    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:28:54.251683 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:54.251590    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:28:54.252102 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:54.251722    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:28:58.114198 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:58.114148    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:28:58.114666 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:58.114606    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:59.252158 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:28:59.252115    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:29:04.252052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:04.252009    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:29:08.114419 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:08.114380    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:29:08.114931 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:08.114521    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:29:08.114931 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:08.114795    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:08.114931 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:08.114912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:29:09.251534 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:09.251481    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:29:11.703076 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.703040    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:29:11.845892 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.845794    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8xrv\" (UniqueName: \"kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv\") pod \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") "
Sep 09 20:29:11.845892 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.845840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location\") pod \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") "
Sep 09 20:29:11.845892 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.845876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert\") pod \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") "
Sep 09 20:29:11.846215 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.845916    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls\") pod \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") "
Sep 09 20:29:11.846215 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.845954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config\") pod \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\" (UID: \"1f3232b0-c14f-41bb-9ba4-f2fddda37939\") "
Sep 09 20:29:11.846303 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.846237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f3232b0-c14f-41bb-9ba4-f2fddda37939" (UID: "1f3232b0-c14f-41bb-9ba4-f2fddda37939"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:11.846370 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.846234    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f3232b0-c14f-41bb-9ba4-f2fddda37939" (UID: "1f3232b0-c14f-41bb-9ba4-f2fddda37939"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:11.846424 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.846386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f3232b0-c14f-41bb-9ba4-f2fddda37939" (UID: "1f3232b0-c14f-41bb-9ba4-f2fddda37939"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:11.848016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.847987    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f3232b0-c14f-41bb-9ba4-f2fddda37939" (UID: "1f3232b0-c14f-41bb-9ba4-f2fddda37939"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:11.848194 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.848176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv" (OuterVolumeSpecName: "kube-api-access-n8xrv") pod "1f3232b0-c14f-41bb-9ba4-f2fddda37939" (UID: "1f3232b0-c14f-41bb-9ba4-f2fddda37939"). InnerVolumeSpecName "kube-api-access-n8xrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:11.856901 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.856866    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerID="f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152" exitCode=0
Sep 09 20:29:11.857015 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.856902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerDied","Data":"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"}
Sep 09 20:29:11.857015 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.856926    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"
Sep 09 20:29:11.857015 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.856962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p" event={"ID":"1f3232b0-c14f-41bb-9ba4-f2fddda37939","Type":"ContainerDied","Data":"bef5591c8f12355e8751cbe0f6ae404ee09f9cde5d10c3b2f49390440cf7fed6"}
Sep 09 20:29:11.857015 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.856993    2564 scope.go:117] "RemoveContainer" containerID="f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"
Sep 09 20:29:11.865897 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.865877    2564 scope.go:117] "RemoveContainer" containerID="634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"
Sep 09 20:29:11.874309 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.874284    2564 scope.go:117] "RemoveContainer" containerID="47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"
Sep 09 20:29:11.881689 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.881651    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:29:11.883551 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.883529    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-6734e-predictor-67f97574b6-pq49p"]
Sep 09 20:29:11.884111 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.884094    2564 scope.go:117] "RemoveContainer" containerID="dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"
Sep 09 20:29:11.892238 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.892217    2564 scope.go:117] "RemoveContainer" containerID="f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"
Sep 09 20:29:11.892528 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:29:11.892510    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152\": container with ID starting with f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152 not found: ID does not exist" containerID="f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"
Sep 09 20:29:11.892569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.892538    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152"} err="failed to get container status \"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152\": rpc error: code = NotFound desc = could not find container \"f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152\": container with ID starting with f890ae9a90794bfa990fac54d840d7a2b0a59835f665cfe02fb5d0e0ffcfd152 not found: ID does not exist"
Sep 09 20:29:11.892569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.892556    2564 scope.go:117] "RemoveContainer" containerID="634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"
Sep 09 20:29:11.892814 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:29:11.892797    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392\": container with ID starting with 634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392 not found: ID does not exist" containerID="634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"
Sep 09 20:29:11.892850 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.892821    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392"} err="failed to get container status \"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392\": rpc error: code = NotFound desc = could not find container \"634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392\": container with ID starting with 634e28d80b6bfc9c64387f0d982ff5152185ae4657e0c40da08bbed099a6c392 not found: ID does not exist"
Sep 09 20:29:11.892850 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.892838    2564 scope.go:117] "RemoveContainer" containerID="47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"
Sep 09 20:29:11.893054 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:29:11.893036    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12\": container with ID starting with 47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12 not found: ID does not exist" containerID="47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"
Sep 09 20:29:11.893115 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.893074    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12"} err="failed to get container status \"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12\": rpc error: code = NotFound desc = could not find container \"47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12\": container with ID starting with 47ccb9c791bdecc8fb21c7677be1d9139f115e108c0a7dc4a173f9bbab61ca12 not found: ID does not exist"
Sep 09 20:29:11.893115 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.893089    2564 scope.go:117] "RemoveContainer" containerID="dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"
Sep 09 20:29:11.893310 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:29:11.893292    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4\": container with ID starting with dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4 not found: ID does not exist" containerID="dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"
Sep 09 20:29:11.893346 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.893332    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4"} err="failed to get container status \"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4\": rpc error: code = NotFound desc = could not find container \"dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4\": container with ID starting with dc086341384a7e506d2aeddb0c50b614a285837701d7d54a07c499ac81e5d1a4 not found: ID does not exist"
Sep 09 20:29:11.946993 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.946955    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:29:11.946993 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.946985    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xrv\" (UniqueName: \"kubernetes.io/projected/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kube-api-access-n8xrv\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:29:11.946993 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.946999    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f3232b0-c14f-41bb-9ba4-f2fddda37939-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:29:11.947243 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.947013    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f3232b0-c14f-41bb-9ba4-f2fddda37939-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:29:11.947243 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:11.947024    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f3232b0-c14f-41bb-9ba4-f2fddda37939-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:29:13.118899 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:29:13.118866    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" path="/var/lib/kubelet/pods/1f3232b0-c14f-41bb-9ba4-f2fddda37939/volumes"
Sep 09 20:30:00.080555 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080521    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080930    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080947    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080968    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kserve-container"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080977    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kserve-container"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.080990    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="storage-initializer"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081000    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="storage-initializer"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081011    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081019    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081031    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081038    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081087    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kube-rbac-proxy"
Sep 09 20:30:00.081185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081098    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kube-rbac-proxy"
Sep 09 20:30:00.081695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081229    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kserve-container"
Sep 09 20:30:00.081695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081244    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kserve-container"
Sep 09 20:30:00.081695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081257    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="agent"
Sep 09 20:30:00.081695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081269    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74395e5f-aca7-4702-b491-38ffd24d6ef2" containerName="kube-rbac-proxy"
Sep 09 20:30:00.081695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.081280    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3232b0-c14f-41bb-9ba4-f2fddda37939" containerName="kube-rbac-proxy"
Sep 09 20:30:00.084748 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.084726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.087341 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087310    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:30:00.087613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087317    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4a4f2-predictor-serving-cert\""
Sep 09 20:30:00.087743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087329    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:30:00.087743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087344    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:30:00.087743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087374    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:30:00.087743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.087480    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-4a4f2-kube-rbac-proxy-sar-config\""
Sep 09 20:30:00.088198 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.088176    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:30:00.094984 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.094955    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:30:00.160125 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.160094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.160317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.160149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.160317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.160179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.160317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.160231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.160317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.160258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dzn\" (UniqueName: \"kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.261628 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.261595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.261835 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.261636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.261835 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.261780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.261977 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.261849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dzn\" (UniqueName: \"kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.261977 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.261961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.262152 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:30:00.262132    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-serving-cert: secret "raw-sklearn-runtime-4a4f2-predictor-serving-cert" not found
Sep 09 20:30:00.262227 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:30:00.262202    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls podName:cc67187c-0187-43b5-b20b-f26d646f6178 nodeName:}" failed. No retries permitted until 2026-09-09 20:30:00.762183368 +0000 UTC m=+1140.136721441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls") pod "raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178") : secret "raw-sklearn-runtime-4a4f2-predictor-serving-cert" not found
Sep 09 20:30:00.262295 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.262228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.262484 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.262463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.262530 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.262503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.271980 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.271945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dzn\" (UniqueName: \"kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.766235 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.766194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.768644 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.768623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") pod \"raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:00.999227 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:00.999186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:30:01.008077 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:01.008042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:01.301623 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:01.301583    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:30:01.308955 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:30:01.308928    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc67187c_0187_43b5_b20b_f26d646f6178.slice/crio-4e18050222e03e4f40098f0c816e9817347060916a667c4de9ace50f261b0fba WatchSource:0}: Error finding container 4e18050222e03e4f40098f0c816e9817347060916a667c4de9ace50f261b0fba: Status 404 returned error can't find the container with id 4e18050222e03e4f40098f0c816e9817347060916a667c4de9ace50f261b0fba
Sep 09 20:30:01.313023 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:01.312999    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:30:02.095557 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:02.095513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerStarted","Data":"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"}
Sep 09 20:30:02.095741 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:02.095613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerStarted","Data":"4e18050222e03e4f40098f0c816e9817347060916a667c4de9ace50f261b0fba"}
Sep 09 20:30:06.117562 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:06.117527    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc67187c-0187-43b5-b20b-f26d646f6178" containerID="05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875" exitCode=0
Sep 09 20:30:06.118050 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:06.117603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerDied","Data":"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"}
Sep 09 20:30:07.123698 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:07.123657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerStarted","Data":"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"}
Sep 09 20:30:07.124155 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:07.123727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerStarted","Data":"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"}
Sep 09 20:30:07.124155 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:07.123934    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:07.144623 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:07.144570    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podStartSLOduration=3.195346999 podStartE2EDuration="7.144550239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.94920324s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:07.143275756 +0000 UTC m=+1146.517813849" watchObservedRunningTime="2026-09-09 20:30:07.144550239 +0000 UTC m=+1146.519088377"
Sep 09 20:30:08.128451 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:08.128405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:08.129616 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:08.129585    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:09.133213 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:09.133173    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:14.137250 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:14.137220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:30:14.137773 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:14.137741    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:24.137900 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:24.137801    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:34.138194 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:34.138149    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:44.138573 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:44.138531    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:30:54.137677 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:30:54.137637    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:31:01.114945 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:01.114912    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:31:01.121405 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:01.121378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:31:01.129937 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:01.129901    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:31:01.132538 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:01.132513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:31:04.137825 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:04.137784    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:31:14.138312 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:14.138272    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:31:24.139194 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:24.139158    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:31:26.944972 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:26.944927    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:31:26.949042 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:26.949019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:26.951116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:26.951091    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:31:26.951249 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:26.951123    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:31:26.959390 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:26.959365    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:31:27.019631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.019591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h826c\" (UniqueName: \"kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.019631 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.019633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.019902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.019663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.019902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.019752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.019902 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.019792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.120658 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.120626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.120832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.120676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.120832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.120708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.120832 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.120735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.120994 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.120850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h826c\" (UniqueName: \"kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.121184 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.121162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.121440 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.121417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.121512 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.121482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.123362 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.123343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.128736 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.128711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h826c\" (UniqueName: \"kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c\") pod \"isvc-canary-create-predictor-5f7b4999bf-nl6vn\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.261708 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.261623    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:27.553283 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:27.553257    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:31:27.555197 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:31:27.555168    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f151ca_f2e7_455b_b69d_03042f1d76e0.slice/crio-fec5b47f72185a3f7293768296b45a60515335944bdfbfc3c733b8077ec13928 WatchSource:0}: Error finding container fec5b47f72185a3f7293768296b45a60515335944bdfbfc3c733b8077ec13928: Status 404 returned error can't find the container with id fec5b47f72185a3f7293768296b45a60515335944bdfbfc3c733b8077ec13928
Sep 09 20:31:28.504143 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:28.504099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerStarted","Data":"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"}
Sep 09 20:31:28.504627 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:28.504171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerStarted","Data":"fec5b47f72185a3f7293768296b45a60515335944bdfbfc3c733b8077ec13928"}
Sep 09 20:31:30.187934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:30.187888    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:31:30.188914 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:30.188875    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" containerID="cri-o://5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6" gracePeriod=30
Sep 09 20:31:30.189934 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:30.189870    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kube-rbac-proxy" containerID="cri-o://ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a" gracePeriod=30
Sep 09 20:31:30.515750 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:30.515662    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc67187c-0187-43b5-b20b-f26d646f6178" containerID="ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a" exitCode=2
Sep 09 20:31:30.515750 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:30.515716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerDied","Data":"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"}
Sep 09 20:31:32.527096 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:32.527041    2564 generic.go:356] "Generic (PLEG): container finished" podID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerID="d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2" exitCode=0
Sep 09 20:31:32.527494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:32.527113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerDied","Data":"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"}
Sep 09 20:31:33.534016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.533978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerStarted","Data":"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"}
Sep 09 20:31:33.534471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.534042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerStarted","Data":"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"}
Sep 09 20:31:33.534471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.534349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:33.534603 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.534507    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:33.535950 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.535922    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:31:33.552040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:33.551990    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podStartSLOduration=3.617923991 podStartE2EDuration="7.551970075s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934046084s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:33.550754028 +0000 UTC m=+1232.925292124" watchObservedRunningTime="2026-09-09 20:31:33.551970075 +0000 UTC m=+1232.926508170"
Sep 09 20:31:34.133896 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:34.133853    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 09 20:31:34.138143 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:34.138104    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:31:34.539769 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:34.539729    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:31:35.441416 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.441389    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:31:35.498569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location\") pod \"cc67187c-0187-43b5-b20b-f26d646f6178\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") "
Sep 09 20:31:35.498569 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5dzn\" (UniqueName: \"kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn\") pod \"cc67187c-0187-43b5-b20b-f26d646f6178\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") "
Sep 09 20:31:35.498823 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498575    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") pod \"cc67187c-0187-43b5-b20b-f26d646f6178\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") "
Sep 09 20:31:35.498823 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert\") pod \"cc67187c-0187-43b5-b20b-f26d646f6178\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") "
Sep 09 20:31:35.498823 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config\") pod \"cc67187c-0187-43b5-b20b-f26d646f6178\" (UID: \"cc67187c-0187-43b5-b20b-f26d646f6178\") "
Sep 09 20:31:35.498823 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc67187c-0187-43b5-b20b-f26d646f6178" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:35.499024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.498907    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc67187c-0187-43b5-b20b-f26d646f6178-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:31:35.499090 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.499020    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc67187c-0187-43b5-b20b-f26d646f6178" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:35.499090 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.499031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc67187c-0187-43b5-b20b-f26d646f6178" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:35.500766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.500741    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn" (OuterVolumeSpecName: "kube-api-access-g5dzn") pod "cc67187c-0187-43b5-b20b-f26d646f6178" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178"). InnerVolumeSpecName "kube-api-access-g5dzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:35.500849 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.500800    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc67187c-0187-43b5-b20b-f26d646f6178" (UID: "cc67187c-0187-43b5-b20b-f26d646f6178"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:35.547046 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.547004    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc67187c-0187-43b5-b20b-f26d646f6178" containerID="5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6" exitCode=0
Sep 09 20:31:35.547558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.547084    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"
Sep 09 20:31:35.547558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.547093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerDied","Data":"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"}
Sep 09 20:31:35.547558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.547184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf" event={"ID":"cc67187c-0187-43b5-b20b-f26d646f6178","Type":"ContainerDied","Data":"4e18050222e03e4f40098f0c816e9817347060916a667c4de9ace50f261b0fba"}
Sep 09 20:31:35.547558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.547220    2564 scope.go:117] "RemoveContainer" containerID="ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"
Sep 09 20:31:35.556486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.556462    2564 scope.go:117] "RemoveContainer" containerID="5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"
Sep 09 20:31:35.564772 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.564753    2564 scope.go:117] "RemoveContainer" containerID="05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"
Sep 09 20:31:35.571132 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.571104    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:31:35.574100 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.574078    2564 scope.go:117] "RemoveContainer" containerID="ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"
Sep 09 20:31:35.574732 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:31:35.574700    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a\": container with ID starting with ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a not found: ID does not exist" containerID="ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"
Sep 09 20:31:35.574861 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.574743    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a"} err="failed to get container status \"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a\": rpc error: code = NotFound desc = could not find container \"ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a\": container with ID starting with ca28238fa6463cd1d6507e45eee41c144d0f0ca6d7b7ad456215c5985cf9119a not found: ID does not exist"
Sep 09 20:31:35.574861 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.574769    2564 scope.go:117] "RemoveContainer" containerID="5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"
Sep 09 20:31:35.575122 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:31:35.575098    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6\": container with ID starting with 5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6 not found: ID does not exist" containerID="5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"
Sep 09 20:31:35.575244 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.575129    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6"} err="failed to get container status \"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6\": rpc error: code = NotFound desc = could not find container \"5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6\": container with ID starting with 5dc5ee376deff8fdf6114550a1b4aced2e49ec3c6cfdd8be666c4f3dc3e6fee6 not found: ID does not exist"
Sep 09 20:31:35.575244 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.575151    2564 scope.go:117] "RemoveContainer" containerID="05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"
Sep 09 20:31:35.575427 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:31:35.575410    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875\": container with ID starting with 05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875 not found: ID does not exist" containerID="05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"
Sep 09 20:31:35.575494 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.575436    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875"} err="failed to get container status \"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875\": rpc error: code = NotFound desc = could not find container \"05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875\": container with ID starting with 05529bc1734f30b7c7a01a56a31bb565a3bccd044ca05abe4e240989276ce875 not found: ID does not exist"
Sep 09 20:31:35.576581 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.576563    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-4a4f2-predictor-858dd47596-z6vrf"]
Sep 09 20:31:35.599917 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.599883    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dzn\" (UniqueName: \"kubernetes.io/projected/cc67187c-0187-43b5-b20b-f26d646f6178-kube-api-access-g5dzn\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:31:35.599917 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.599912    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc67187c-0187-43b5-b20b-f26d646f6178-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:31:35.599917 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.599923    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:31:35.600196 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:35.599931    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc67187c-0187-43b5-b20b-f26d646f6178-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:31:37.118571 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:37.118536    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" path="/var/lib/kubelet/pods/cc67187c-0187-43b5-b20b-f26d646f6178/volumes"
Sep 09 20:31:39.544729 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:39.544693    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:31:39.545249 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:39.545197    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:31:49.545938 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:49.545885    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:31:59.546327 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:31:59.546223    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:09.546007 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:09.545962    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:19.545567 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:19.545527    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:29.546166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:29.546122    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:39.546075 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:39.546013    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:43.119334 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:43.119296    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:32:49.737442 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.737408    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:32:49.737883 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.737659    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container" containerID="cri-o://68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95" gracePeriod=30
Sep 09 20:32:49.737883 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.737698    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kube-rbac-proxy" containerID="cri-o://0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6" gracePeriod=30
Sep 09 20:32:49.738572 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.738186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:32:49.909199 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.909165    2564 generic.go:356] "Generic (PLEG): container finished" podID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerID="0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6" exitCode=2
Sep 09 20:32:49.909378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:49.909243    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerDied","Data":"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"}
Sep 09 20:32:54.540230 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.540180    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 09 20:32:54.789301 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.789275    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:32:54.897784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.897689    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config\") pod \"00f151ca-f2e7-455b-b69d-03042f1d76e0\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") "
Sep 09 20:32:54.897784 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.897767    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location\") pod \"00f151ca-f2e7-455b-b69d-03042f1d76e0\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") "
Sep 09 20:32:54.897981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.897807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls\") pod \"00f151ca-f2e7-455b-b69d-03042f1d76e0\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") "
Sep 09 20:32:54.897981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.897830    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h826c\" (UniqueName: \"kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c\") pod \"00f151ca-f2e7-455b-b69d-03042f1d76e0\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") "
Sep 09 20:32:54.897981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.897895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert\") pod \"00f151ca-f2e7-455b-b69d-03042f1d76e0\" (UID: \"00f151ca-f2e7-455b-b69d-03042f1d76e0\") "
Sep 09 20:32:54.898179 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.898118    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00f151ca-f2e7-455b-b69d-03042f1d76e0" (UID: "00f151ca-f2e7-455b-b69d-03042f1d76e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:54.898242 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.898175    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00f151ca-f2e7-455b-b69d-03042f1d76e0" (UID: "00f151ca-f2e7-455b-b69d-03042f1d76e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:54.898353 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.898331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00f151ca-f2e7-455b-b69d-03042f1d76e0" (UID: "00f151ca-f2e7-455b-b69d-03042f1d76e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:54.900001 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.899979    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00f151ca-f2e7-455b-b69d-03042f1d76e0" (UID: "00f151ca-f2e7-455b-b69d-03042f1d76e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:54.900074 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.900037    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c" (OuterVolumeSpecName: "kube-api-access-h826c") pod "00f151ca-f2e7-455b-b69d-03042f1d76e0" (UID: "00f151ca-f2e7-455b-b69d-03042f1d76e0"). InnerVolumeSpecName "kube-api-access-h826c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:54.936376 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.936342    2564 generic.go:356] "Generic (PLEG): container finished" podID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerID="68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95" exitCode=0
Sep 09 20:32:54.936536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.936401    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"
Sep 09 20:32:54.936536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.936421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerDied","Data":"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"}
Sep 09 20:32:54.936536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.936491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn" event={"ID":"00f151ca-f2e7-455b-b69d-03042f1d76e0","Type":"ContainerDied","Data":"fec5b47f72185a3f7293768296b45a60515335944bdfbfc3c733b8077ec13928"}
Sep 09 20:32:54.936536 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.936529    2564 scope.go:117] "RemoveContainer" containerID="0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"
Sep 09 20:32:54.948472 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.948444    2564 scope.go:117] "RemoveContainer" containerID="68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"
Sep 09 20:32:54.957581 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.957560    2564 scope.go:117] "RemoveContainer" containerID="d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"
Sep 09 20:32:54.959378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.959353    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:32:54.963765 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.963741    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-nl6vn"]
Sep 09 20:32:54.966441 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.966422    2564 scope.go:117] "RemoveContainer" containerID="0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"
Sep 09 20:32:54.966755 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:32:54.966732    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6\": container with ID starting with 0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6 not found: ID does not exist" containerID="0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"
Sep 09 20:32:54.966835 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.966765    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6"} err="failed to get container status \"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6\": rpc error: code = NotFound desc = could not find container \"0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6\": container with ID starting with 0db4912808854f9521576f93b10d69bf537479064e4d1ad0ce0cac0137031da6 not found: ID does not exist"
Sep 09 20:32:54.966835 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.966785    2564 scope.go:117] "RemoveContainer" containerID="68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"
Sep 09 20:32:54.967079 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:32:54.967047    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95\": container with ID starting with 68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95 not found: ID does not exist" containerID="68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"
Sep 09 20:32:54.967132 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.967083    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95"} err="failed to get container status \"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95\": rpc error: code = NotFound desc = could not find container \"68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95\": container with ID starting with 68571557e7dfc78ae6a1f5e1a232c61a17b4cdd830c2b8db7474237923d13b95 not found: ID does not exist"
Sep 09 20:32:54.967132 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.967098    2564 scope.go:117] "RemoveContainer" containerID="d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"
Sep 09 20:32:54.967368 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:32:54.967348    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2\": container with ID starting with d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2 not found: ID does not exist" containerID="d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"
Sep 09 20:32:54.967410 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.967375    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2"} err="failed to get container status \"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2\": rpc error: code = NotFound desc = could not find container \"d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2\": container with ID starting with d5cc0928c2138a72b111b2400ae2d43dfe1dbfedb313802c3b6fa6679ccf84c2 not found: ID does not exist"
Sep 09 20:32:54.998737 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.998696    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h826c\" (UniqueName: \"kubernetes.io/projected/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-api-access-h826c\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:32:54.998737 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.998723    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:32:54.998737 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.998735    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00f151ca-f2e7-455b-b69d-03042f1d76e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:32:54.998737 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.998748    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00f151ca-f2e7-455b-b69d-03042f1d76e0-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:32:54.999110 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:54.998762    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00f151ca-f2e7-455b-b69d-03042f1d76e0-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:32:55.118833 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:32:55.118801    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" path="/var/lib/kubelet/pods/00f151ca-f2e7-455b-b69d-03042f1d76e0/volumes"
Sep 09 20:33:02.172531 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.172490    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:33:02.173161 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173136    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container"
Sep 09 20:33:02.173161 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173164    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173191    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kube-rbac-proxy"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173201    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kube-rbac-proxy"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173219    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173229    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173250    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kube-rbac-proxy"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173259    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kube-rbac-proxy"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173285    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="storage-initializer"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173293    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="storage-initializer"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173302    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="storage-initializer"
Sep 09 20:33:02.173329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173310    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="storage-initializer"
Sep 09 20:33:02.173788 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173430    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kube-rbac-proxy"
Sep 09 20:33:02.173788 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173445    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kserve-container"
Sep 09 20:33:02.173788 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173455    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f151ca-f2e7-455b-b69d-03042f1d76e0" containerName="kserve-container"
Sep 09 20:33:02.173788 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.173464    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc67187c-0187-43b5-b20b-f26d646f6178" containerName="kube-rbac-proxy"
Sep 09 20:33:02.181087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.179315    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:33:02.181480 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.181441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.184254 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.184221    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:33:02.184254 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.184236    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:33:02.184437 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.184272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:33:02.184437 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.184232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:33:02.185027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.185008    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:33:02.185309 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.185292    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:33:02.185428 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.185412    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:33:02.185506 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.185458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.187867 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.187838    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:33:02.189957 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.189932    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:33:02.192968 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.192938    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:33:02.254516 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.254516 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9x4\" (UniqueName: \"kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcqr\" (UniqueName: \"kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.254755 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.254730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.355303 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.355490 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.355490 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.355490 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.355490 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:33:02.355417    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:33:02.355512    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls podName:a1e901d8-2685-4c92-a85c-a94b915a272c nodeName:}" failed. No retries permitted until 2026-09-09 20:33:02.855490723 +0000 UTC m=+1322.230028809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls") pod "isvc-canary-promote-predictor-648d7cfc8f-qnrvf" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcqr\" (UniqueName: \"kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.355739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.356077 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9x4\" (UniqueName: \"kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.356077 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.355826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.356252 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.356226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.356356 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.356233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.356356 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.356232    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.356436 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.356381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.356436 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.356407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.358136 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.358116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.363658 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.363632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcqr\" (UniqueName: \"kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.363993 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.363973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9x4\" (UniqueName: \"kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.498407 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.498311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:02.772856 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.772793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:33:02.775151 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:33:02.775122    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179983a5_6524_41f0_a3f4_6a09b717c43c.slice/crio-0e3dca075cfc827d3b0430453917ccb5b8ccb93ab018bfe32f8731568ee1b0c8 WatchSource:0}: Error finding container 0e3dca075cfc827d3b0430453917ccb5b8ccb93ab018bfe32f8731568ee1b0c8: Status 404 returned error can't find the container with id 0e3dca075cfc827d3b0430453917ccb5b8ccb93ab018bfe32f8731568ee1b0c8
Sep 09 20:33:02.859444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.859397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.861881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.861846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-qnrvf\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:02.979240 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.979198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerStarted","Data":"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"}
Sep 09 20:33:02.979402 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:02.979255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerStarted","Data":"0e3dca075cfc827d3b0430453917ccb5b8ccb93ab018bfe32f8731568ee1b0c8"}
Sep 09 20:33:03.107124 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:03.107089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:03.386430 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:03.386401    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:33:03.389572 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:33:03.389540    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e901d8_2685_4c92_a85c_a94b915a272c.slice/crio-c99bcf523a303eec45f3e4eb17d0c162973c6003637694bfa2c0714db12e6957 WatchSource:0}: Error finding container c99bcf523a303eec45f3e4eb17d0c162973c6003637694bfa2c0714db12e6957: Status 404 returned error can't find the container with id c99bcf523a303eec45f3e4eb17d0c162973c6003637694bfa2c0714db12e6957
Sep 09 20:33:03.986099 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:03.986034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerStarted","Data":"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"}
Sep 09 20:33:03.986263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:03.986137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerStarted","Data":"c99bcf523a303eec45f3e4eb17d0c162973c6003637694bfa2c0714db12e6957"}
Sep 09 20:33:07.000818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:07.000722    2564 generic.go:356] "Generic (PLEG): container finished" podID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerID="c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49" exitCode=0
Sep 09 20:33:07.000818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:07.000776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerDied","Data":"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"}
Sep 09 20:33:08.007881 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.007800    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerID="3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83" exitCode=0
Sep 09 20:33:08.008306 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.007879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerDied","Data":"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"}
Sep 09 20:33:08.010435 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.010410    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerStarted","Data":"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"}
Sep 09 20:33:08.010538 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.010469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerStarted","Data":"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"}
Sep 09 20:33:08.010690 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.010671    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:08.043553 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:08.043501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podStartSLOduration=2.138286543 podStartE2EDuration="6.043483084s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905196541s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:08.04147825 +0000 UTC m=+1327.416016345" watchObservedRunningTime="2026-09-09 20:33:08.043483084 +0000 UTC m=+1327.418021177"
Sep 09 20:33:09.017968 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.017930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerStarted","Data":"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"}
Sep 09 20:33:09.018454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.018001    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerStarted","Data":"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"}
Sep 09 20:33:09.018454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.018036    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:09.018454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.018440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:09.019747 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.019722    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:09.037803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:09.037740    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podStartSLOduration=2.875528593 podStartE2EDuration="7.037725175s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.162196582s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:09.035006134 +0000 UTC m=+1328.409544230" watchObservedRunningTime="2026-09-09 20:33:09.037725175 +0000 UTC m=+1328.412263266"
Sep 09 20:33:10.025938 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:10.025894    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:10.026427 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:10.026121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:10.027343 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:10.027317    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:11.031945 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:11.031904    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:15.031289 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:15.031257    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:33:15.031921 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:15.031892    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:16.036720 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:16.036690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:33:16.037217 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:16.037189    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:25.032554 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:25.032458    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:26.037701 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:26.037660    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:35.032591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:35.032543    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:36.037691 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:36.037638    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:45.031879 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:45.031839    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:46.037887 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:46.037842    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:33:55.032829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:55.032777    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:33:56.038123 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:33:56.038080    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:34:05.032744 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:05.032700    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:34:06.038229 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:06.038180    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:34:15.032460 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:15.032414    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:34:16.038016 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:16.037970    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:34:17.113577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:17.113532    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:34:25.033087 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.033031    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:34:25.164133 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.164096    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:34:25.164458 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.164420    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container" containerID="cri-o://3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a" gracePeriod=30
Sep 09 20:34:25.164458 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.164434    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kube-rbac-proxy" containerID="cri-o://e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6" gracePeriod=30
Sep 09 20:34:25.165347 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.165197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:34:25.412645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.412553    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerID="e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6" exitCode=2
Sep 09 20:34:25.412645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:25.412622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerDied","Data":"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"}
Sep 09 20:34:26.032665 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:26.032624    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 09 20:34:30.311682 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.311655    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:34:30.432978 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.432891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config\") pod \"a1e901d8-2685-4c92-a85c-a94b915a272c\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") "
Sep 09 20:34:30.432978 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.432967    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") pod \"a1e901d8-2685-4c92-a85c-a94b915a272c\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") "
Sep 09 20:34:30.433340 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.432990    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert\") pod \"a1e901d8-2685-4c92-a85c-a94b915a272c\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") "
Sep 09 20:34:30.433340 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.433032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kcqr\" (UniqueName: \"kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr\") pod \"a1e901d8-2685-4c92-a85c-a94b915a272c\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") "
Sep 09 20:34:30.433340 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.433076    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location\") pod \"a1e901d8-2685-4c92-a85c-a94b915a272c\" (UID: \"a1e901d8-2685-4c92-a85c-a94b915a272c\") "
Sep 09 20:34:30.433506 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.433442    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1e901d8-2685-4c92-a85c-a94b915a272c" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:30.433506 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.433465    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1e901d8-2685-4c92-a85c-a94b915a272c" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:30.433577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.433545    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1e901d8-2685-4c92-a85c-a94b915a272c" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:30.435270 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.435235    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1e901d8-2685-4c92-a85c-a94b915a272c" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:30.435391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.435269    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr" (OuterVolumeSpecName: "kube-api-access-5kcqr") pod "a1e901d8-2685-4c92-a85c-a94b915a272c" (UID: "a1e901d8-2685-4c92-a85c-a94b915a272c"). InnerVolumeSpecName "kube-api-access-5kcqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:30.438236 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.438208    2564 generic.go:356] "Generic (PLEG): container finished" podID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerID="3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a" exitCode=0
Sep 09 20:34:30.438365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.438261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerDied","Data":"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"}
Sep 09 20:34:30.438365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.438273    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"
Sep 09 20:34:30.438365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.438342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf" event={"ID":"a1e901d8-2685-4c92-a85c-a94b915a272c","Type":"ContainerDied","Data":"c99bcf523a303eec45f3e4eb17d0c162973c6003637694bfa2c0714db12e6957"}
Sep 09 20:34:30.438520 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.438371    2564 scope.go:117] "RemoveContainer" containerID="e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"
Sep 09 20:34:30.456275 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.456257    2564 scope.go:117] "RemoveContainer" containerID="3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"
Sep 09 20:34:30.465483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.465462    2564 scope.go:117] "RemoveContainer" containerID="3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"
Sep 09 20:34:30.467378 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.467356    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:34:30.470811 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.470789    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-qnrvf"]
Sep 09 20:34:30.474316 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.474299    2564 scope.go:117] "RemoveContainer" containerID="e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"
Sep 09 20:34:30.474609 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:34:30.474588    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6\": container with ID starting with e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6 not found: ID does not exist" containerID="e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"
Sep 09 20:34:30.474685 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.474623    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6"} err="failed to get container status \"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6\": rpc error: code = NotFound desc = could not find container \"e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6\": container with ID starting with e018e4c9ffd76487bc807060f25dc49b1c938c6ab19d16703ed24904a9baccb6 not found: ID does not exist"
Sep 09 20:34:30.474685 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.474648    2564 scope.go:117] "RemoveContainer" containerID="3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"
Sep 09 20:34:30.474907 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:34:30.474891    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a\": container with ID starting with 3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a not found: ID does not exist" containerID="3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"
Sep 09 20:34:30.474950 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.474914    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a"} err="failed to get container status \"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a\": rpc error: code = NotFound desc = could not find container \"3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a\": container with ID starting with 3e52f2cf6f8056930d96f3cf8cf05a048249e83ac252cc7272f76b9fc4ce6d0a not found: ID does not exist"
Sep 09 20:34:30.474950 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.474929    2564 scope.go:117] "RemoveContainer" containerID="3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"
Sep 09 20:34:30.475166 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:34:30.475150    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83\": container with ID starting with 3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83 not found: ID does not exist" containerID="3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"
Sep 09 20:34:30.475219 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.475171    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83"} err="failed to get container status \"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83\": rpc error: code = NotFound desc = could not find container \"3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83\": container with ID starting with 3048d44c6c85d45cc3efbfff2ba986321f8dd04353ed72b5254222e15d089e83 not found: ID does not exist"
Sep 09 20:34:30.533725 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.533686    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.533725 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.533720    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1e901d8-2685-4c92-a85c-a94b915a272c-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.533725 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.533732    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1e901d8-2685-4c92-a85c-a94b915a272c-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.533959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.533741    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcqr\" (UniqueName: \"kubernetes.io/projected/a1e901d8-2685-4c92-a85c-a94b915a272c-kube-api-access-5kcqr\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:34:30.533959 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:30.533750    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1e901d8-2685-4c92-a85c-a94b915a272c-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:34:31.119799 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:34:31.119768    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" path="/var/lib/kubelet/pods/a1e901d8-2685-4c92-a85c-a94b915a272c/volumes"
Sep 09 20:36:01.155212 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:01.155184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:36:01.159825 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:01.159802    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:36:01.161645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:01.161625    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:36:01.166215 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:01.166193    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:36:02.486231 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:02.486187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:36:02.486733 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:02.486585    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" containerID="cri-o://12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe" gracePeriod=30
Sep 09 20:36:02.486733 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:02.486607    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kube-rbac-proxy" containerID="cri-o://9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de" gracePeriod=30
Sep 09 20:36:02.879757 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:02.879721    2564 generic.go:356] "Generic (PLEG): container finished" podID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerID="9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de" exitCode=2
Sep 09 20:36:02.879955 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:02.879793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerDied","Data":"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"}
Sep 09 20:36:05.026322 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:05.026276    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 09 20:36:05.031814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:05.031782    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 09 20:36:07.543070 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.543031    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:36:07.657938 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.657838    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config\") pod \"179983a5-6524-41f0-a3f4-6a09b717c43c\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") "
Sep 09 20:36:07.657938 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.657890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert\") pod \"179983a5-6524-41f0-a3f4-6a09b717c43c\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") "
Sep 09 20:36:07.657938 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.657912    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls\") pod \"179983a5-6524-41f0-a3f4-6a09b717c43c\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") "
Sep 09 20:36:07.658258 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658029    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location\") pod \"179983a5-6524-41f0-a3f4-6a09b717c43c\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") "
Sep 09 20:36:07.658258 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658126    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w9x4\" (UniqueName: \"kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4\") pod \"179983a5-6524-41f0-a3f4-6a09b717c43c\" (UID: \"179983a5-6524-41f0-a3f4-6a09b717c43c\") "
Sep 09 20:36:07.658365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658285    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "179983a5-6524-41f0-a3f4-6a09b717c43c" (UID: "179983a5-6524-41f0-a3f4-6a09b717c43c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:07.658365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658298    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "179983a5-6524-41f0-a3f4-6a09b717c43c" (UID: "179983a5-6524-41f0-a3f4-6a09b717c43c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:07.658365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "179983a5-6524-41f0-a3f4-6a09b717c43c" (UID: "179983a5-6524-41f0-a3f4-6a09b717c43c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:07.658471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658405    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.658471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658425    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/179983a5-6524-41f0-a3f4-6a09b717c43c-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.658471 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.658438    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179983a5-6524-41f0-a3f4-6a09b717c43c-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.659970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.659951    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "179983a5-6524-41f0-a3f4-6a09b717c43c" (UID: "179983a5-6524-41f0-a3f4-6a09b717c43c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:07.660211 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.660194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4" (OuterVolumeSpecName: "kube-api-access-4w9x4") pod "179983a5-6524-41f0-a3f4-6a09b717c43c" (UID: "179983a5-6524-41f0-a3f4-6a09b717c43c"). InnerVolumeSpecName "kube-api-access-4w9x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:07.759559 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.759517    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/179983a5-6524-41f0-a3f4-6a09b717c43c-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.759559 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.759550    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9x4\" (UniqueName: \"kubernetes.io/projected/179983a5-6524-41f0-a3f4-6a09b717c43c-kube-api-access-4w9x4\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.906486 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.906450    2564 generic.go:356] "Generic (PLEG): container finished" podID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerID="12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe" exitCode=0
Sep 09 20:36:07.906648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.906490    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerDied","Data":"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"}
Sep 09 20:36:07.906648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.906506    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"
Sep 09 20:36:07.906648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.906544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm" event={"ID":"179983a5-6524-41f0-a3f4-6a09b717c43c","Type":"ContainerDied","Data":"0e3dca075cfc827d3b0430453917ccb5b8ccb93ab018bfe32f8731568ee1b0c8"}
Sep 09 20:36:07.906648 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.906572    2564 scope.go:117] "RemoveContainer" containerID="9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"
Sep 09 20:36:07.923172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.923139    2564 scope.go:117] "RemoveContainer" containerID="12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"
Sep 09 20:36:07.931790 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.931763    2564 scope.go:117] "RemoveContainer" containerID="c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"
Sep 09 20:36:07.932871 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.932804    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:36:07.938651 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.938625    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-q9zzm"]
Sep 09 20:36:07.940444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.940428    2564 scope.go:117] "RemoveContainer" containerID="9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"
Sep 09 20:36:07.940730 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:07.940710    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de\": container with ID starting with 9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de not found: ID does not exist" containerID="9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"
Sep 09 20:36:07.940774 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.940741    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de"} err="failed to get container status \"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de\": rpc error: code = NotFound desc = could not find container \"9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de\": container with ID starting with 9658e8b9b49a7e5aa1a593d068299f2782f31907f4010e53aff7a31f99a954de not found: ID does not exist"
Sep 09 20:36:07.940774 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.940761    2564 scope.go:117] "RemoveContainer" containerID="12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"
Sep 09 20:36:07.941034 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:07.941015    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe\": container with ID starting with 12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe not found: ID does not exist" containerID="12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"
Sep 09 20:36:07.941123 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.941039    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe"} err="failed to get container status \"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe\": rpc error: code = NotFound desc = could not find container \"12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe\": container with ID starting with 12c1be6e51d38269fa237d56ae4a4688b1492d591ceab9d33f9ba9b879a184fe not found: ID does not exist"
Sep 09 20:36:07.941123 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.941077    2564 scope.go:117] "RemoveContainer" containerID="c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"
Sep 09 20:36:07.941346 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:07.941329    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49\": container with ID starting with c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49 not found: ID does not exist" containerID="c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"
Sep 09 20:36:07.941386 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:07.941352    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49"} err="failed to get container status \"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49\": rpc error: code = NotFound desc = could not find container \"c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49\": container with ID starting with c6e09c2c601c6a04d3ff05b7240dfbcacf22d23262fef348c7c0126f23f5ba49 not found: ID does not exist"
Sep 09 20:36:09.119445 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:09.119410    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" path="/var/lib/kubelet/pods/179983a5-6524-41f0-a3f4-6a09b717c43c/volumes"
Sep 09 20:36:16.628729 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.628688    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:36:16.629288 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629267    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="storage-initializer"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629290    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="storage-initializer"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629312    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629320    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629334    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="storage-initializer"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629343    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="storage-initializer"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629352    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629360    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.629379 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629377    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629385    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629409    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629420    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629529    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629545    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kserve-container"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629556    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="179983a5-6524-41f0-a3f4-6a09b717c43c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.629771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.629565    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e901d8-2685-4c92-a85c-a94b915a272c" containerName="kube-rbac-proxy"
Sep 09 20:36:16.634438 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.634415    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.636578 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.636557    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 09 20:36:16.636706 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.636610    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 20:36:16.636706 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.636667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:36:16.636706 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.636700    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:36:16.637616 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.637552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:36:16.639751 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.639727    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:36:16.738735 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.738696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.738955 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.738756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmb98\" (UniqueName: \"kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.738955 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.738840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.840024 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.839988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.840230 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.840090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.840230 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.840148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmb98\" (UniqueName: \"kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.840230 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:16.840177    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 09 20:36:16.840356 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:16.840260    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls podName:f5ef2777-6ecf-4055-9af9-81a08f1d685e nodeName:}" failed. No retries permitted until 2026-09-09 20:36:17.340238457 +0000 UTC m=+1516.714776543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" (UID: "f5ef2777-6ecf-4055-9af9-81a08f1d685e") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 09 20:36:16.840732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.840710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:16.848482 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:16.848458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmb98\" (UniqueName: \"kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:17.344739 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.344698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:17.347244 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.347214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-c6rpv\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:17.546463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.546422    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:17.817117 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.817004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:36:17.819569 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:36:17.819541    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ef2777_6ecf_4055_9af9_81a08f1d685e.slice/crio-2167bb0e2b17de2a735999568daffc6f867336591d5c6907348a5fa2bd59cc46 WatchSource:0}: Error finding container 2167bb0e2b17de2a735999568daffc6f867336591d5c6907348a5fa2bd59cc46: Status 404 returned error can't find the container with id 2167bb0e2b17de2a735999568daffc6f867336591d5c6907348a5fa2bd59cc46
Sep 09 20:36:17.958676 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.958647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerStarted","Data":"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"}
Sep 09 20:36:17.958814 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:17.958702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerStarted","Data":"2167bb0e2b17de2a735999568daffc6f867336591d5c6907348a5fa2bd59cc46"}
Sep 09 20:36:18.965777 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:18.965733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerStarted","Data":"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"}
Sep 09 20:36:18.966264 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:18.965859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:18.983262 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:18.982871    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" podStartSLOduration=2.982851134 podStartE2EDuration="2.982851134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:18.981560634 +0000 UTC m=+1518.356098742" watchObservedRunningTime="2026-09-09 20:36:18.982851134 +0000 UTC m=+1518.357389228"
Sep 09 20:36:19.970791 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:19.970755    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:19.972663 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:19.972642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:26.985505 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:26.985417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:36:36.671468 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.671428    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:36:36.675395 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.675373    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.677747 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.677721    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 09 20:36:36.677747 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.677742    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 20:36:36.677747 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.677751    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:36:36.677997 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.677806    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:36:36.685029 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.685006    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:36:36.721257 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.721225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.721629 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.721293    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.721629 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.721334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.721629 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.721356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjvr\" (UniqueName: \"kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.721629 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.721444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822392 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.822355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822590 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.822407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822590 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.822453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822590 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.822484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822590 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:36.822574    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 09 20:36:36.822771 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.822585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjvr\" (UniqueName: \"kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.822771 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:36:36.822665    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls podName:bc30d9b7-6e07-4726-8f48-e83456127b6e nodeName:}" failed. No retries permitted until 2026-09-09 20:36:37.322643197 +0000 UTC m=+1536.697181285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls") pod "isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 09 20:36:36.823128 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.823106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.823183 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.823168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.823244 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.823227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:36.831147 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:36.831124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjvr\" (UniqueName: \"kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:37.326387 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:37.326341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:37.328847 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:37.328825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-69444974bf-rrtdn\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:37.586720 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:37.586614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:37.862185 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:37.862157    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:36:37.864044 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:36:37.864005    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc30d9b7_6e07_4726_8f48_e83456127b6e.slice/crio-bab29165fd57ac1249bee3743897ced53131808b7f24fe6067a6906dbf381551 WatchSource:0}: Error finding container bab29165fd57ac1249bee3743897ced53131808b7f24fe6067a6906dbf381551: Status 404 returned error can't find the container with id bab29165fd57ac1249bee3743897ced53131808b7f24fe6067a6906dbf381551
Sep 09 20:36:38.061598 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:38.061555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerStarted","Data":"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"}
Sep 09 20:36:38.061806 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:38.061629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerStarted","Data":"bab29165fd57ac1249bee3743897ced53131808b7f24fe6067a6906dbf381551"}
Sep 09 20:36:42.080844 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:42.080807    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerID="df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380" exitCode=0
Sep 09 20:36:42.081284 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:42.080892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerDied","Data":"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"}
Sep 09 20:36:43.089007 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:43.088973    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerStarted","Data":"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"}
Sep 09 20:36:43.089404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:43.089039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerStarted","Data":"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"}
Sep 09 20:36:43.089404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:43.089080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerStarted","Data":"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"}
Sep 09 20:36:43.089404 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:43.089298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:43.109645 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:43.109572    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podStartSLOduration=3.422572671 podStartE2EDuration="7.109554758s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.686982087s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:43.107897086 +0000 UTC m=+1542.482435181" watchObservedRunningTime="2026-09-09 20:36:43.109554758 +0000 UTC m=+1542.484092851"
Sep 09 20:36:44.094412 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:44.094374    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:44.094865 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:44.094442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:44.095933 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:44.095897    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:36:44.096537 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:44.096513    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:45.099369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:45.099328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:36:45.099847 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:45.099825    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:50.103797 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:50.103761    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:36:50.106191 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:50.104465    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:36:50.106191 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:36:50.104753    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:00.104558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:00.104510    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:00.105037 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:00.104912    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:10.104794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:10.104734    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:10.105336 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:10.105129    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:20.104558 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:20.104507    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:20.105000 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:20.104902    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:30.104743 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:30.104701    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:30.105251 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:30.105185    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:40.105361 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:40.105314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:40.105805 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:40.105711    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:50.104577 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:50.104529    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:37:50.105106 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:37:50.104902    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:00.105281 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:00.105202    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:00.105727 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:00.105301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:11.751247 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.751216    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-c6rpv_f5ef2777-6ecf-4055-9af9-81a08f1d685e/kserve-container/0.log"
Sep 09 20:38:11.930610 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.930574    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:38:11.930903 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.930855    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" containerID="cri-o://26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e" gracePeriod=30
Sep 09 20:38:11.930978 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.930891    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" containerID="cri-o://41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3" gracePeriod=30
Sep 09 20:38:11.931052 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.930891    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" containerID="cri-o://06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c" gracePeriod=30
Sep 09 20:38:11.957711 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.957682    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:38:11.957929 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.957905    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kserve-container" containerID="cri-o://88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e" gracePeriod=30
Sep 09 20:38:11.958040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.957995    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kube-rbac-proxy" containerID="cri-o://9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3" gracePeriod=30
Sep 09 20:38:11.981748 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:11.981718    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 09 20:38:12.218787 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.218761    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:38:12.339444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.339348    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmb98\" (UniqueName: \"kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98\") pod \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") "
Sep 09 20:38:12.339444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.339415    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config\") pod \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") "
Sep 09 20:38:12.339444 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.339434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") pod \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\" (UID: \"f5ef2777-6ecf-4055-9af9-81a08f1d685e\") "
Sep 09 20:38:12.339878 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.339839    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5ef2777-6ecf-4055-9af9-81a08f1d685e" (UID: "f5ef2777-6ecf-4055-9af9-81a08f1d685e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:12.341617 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.341590    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5ef2777-6ecf-4055-9af9-81a08f1d685e" (UID: "f5ef2777-6ecf-4055-9af9-81a08f1d685e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:12.341713 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.341671    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98" (OuterVolumeSpecName: "kube-api-access-zmb98") pod "f5ef2777-6ecf-4055-9af9-81a08f1d685e" (UID: "f5ef2777-6ecf-4055-9af9-81a08f1d685e"). InnerVolumeSpecName "kube-api-access-zmb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:12.440165 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.440133    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmb98\" (UniqueName: \"kubernetes.io/projected/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-api-access-zmb98\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:12.440165 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.440162    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef2777-6ecf-4055-9af9-81a08f1d685e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:12.440165 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.440172    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef2777-6ecf-4055-9af9-81a08f1d685e-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:12.542781 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.542746    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerID="06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c" exitCode=2
Sep 09 20:38:12.542970 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.542819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerDied","Data":"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"}
Sep 09 20:38:12.544591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544565    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerID="9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3" exitCode=2
Sep 09 20:38:12.544591 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544587    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerID="88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e" exitCode=2
Sep 09 20:38:12.544795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerDied","Data":"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"}
Sep 09 20:38:12.544795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544622    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"
Sep 09 20:38:12.544795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerDied","Data":"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"}
Sep 09 20:38:12.544795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544671    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv" event={"ID":"f5ef2777-6ecf-4055-9af9-81a08f1d685e","Type":"ContainerDied","Data":"2167bb0e2b17de2a735999568daffc6f867336591d5c6907348a5fa2bd59cc46"}
Sep 09 20:38:12.544795 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.544695    2564 scope.go:117] "RemoveContainer" containerID="9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"
Sep 09 20:38:12.557415 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.557393    2564 scope.go:117] "RemoveContainer" containerID="88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"
Sep 09 20:38:12.565932 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.565800    2564 scope.go:117] "RemoveContainer" containerID="9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"
Sep 09 20:38:12.566406 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:12.566370    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3\": container with ID starting with 9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3 not found: ID does not exist" containerID="9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"
Sep 09 20:38:12.566523 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.566417    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"} err="failed to get container status \"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3\": rpc error: code = NotFound desc = could not find container \"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3\": container with ID starting with 9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3 not found: ID does not exist"
Sep 09 20:38:12.566523 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.566443    2564 scope.go:117] "RemoveContainer" containerID="88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"
Sep 09 20:38:12.566787 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:12.566769    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e\": container with ID starting with 88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e not found: ID does not exist" containerID="88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"
Sep 09 20:38:12.566844 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.566793    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"} err="failed to get container status \"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e\": rpc error: code = NotFound desc = could not find container \"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e\": container with ID starting with 88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e not found: ID does not exist"
Sep 09 20:38:12.566844 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.566811    2564 scope.go:117] "RemoveContainer" containerID="9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"
Sep 09 20:38:12.567116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.567099    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3"} err="failed to get container status \"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3\": rpc error: code = NotFound desc = could not find container \"9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3\": container with ID starting with 9712876f5d1cb31e259f45d46602327e700b0bedd74d4a24632da96482a0caa3 not found: ID does not exist"
Sep 09 20:38:12.567116 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.567115    2564 scope.go:117] "RemoveContainer" containerID="88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"
Sep 09 20:38:12.567350 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.567334    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e"} err="failed to get container status \"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e\": rpc error: code = NotFound desc = could not find container \"88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e\": container with ID starting with 88cdbc43ee9c007eb463adf426a99646b4ba53a6bd384d850006573f410e847e not found: ID does not exist"
Sep 09 20:38:12.568385 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.568367    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:38:12.571760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:12.571739    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-c6rpv"]
Sep 09 20:38:13.119579 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:13.119545    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" path="/var/lib/kubelet/pods/f5ef2777-6ecf-4055-9af9-81a08f1d685e/volumes"
Sep 09 20:38:15.099847 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:15.099801    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:17.578717 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:17.578682    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerID="26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e" exitCode=0
Sep 09 20:38:17.579183 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:17.578746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerDied","Data":"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"}
Sep 09 20:38:20.100235 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:20.100189    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:20.104518 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:20.104485    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:38:20.104804 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:20.104781    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:25.100013 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:25.099958    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:25.100424 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:25.100152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:28.026320 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.026290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:28.047935 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.047902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:28.056649 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.056600    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:28.065940 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.065917    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:28.842026 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.841995    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:28.852260 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.852228    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:28.861184 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.861159    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:28.869125 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:28.869085    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:29.668982 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:29.668920    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:29.678949 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:29.678926    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:29.687566 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:29.687544    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:29.697332 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:29.697310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:30.099907 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.099869    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:30.105301 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.105263    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:38:30.105602 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.105578    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:30.458086 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.457975    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:30.469548 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.469523    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:30.478803 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.478767    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:30.487926 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:30.487904    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:31.286638 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:31.286596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:31.297882 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:31.297852    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:31.307492 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:31.307470    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:31.316587 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:31.316561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:32.111704 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.111671    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:32.121545 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.121514    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:32.130216 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.130182    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:32.138702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.138667    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:32.918782 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.918747    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:32.929351 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.929320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:32.938073 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.938034    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:32.947674 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:32.947647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:33.743901 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:33.743874    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:33.755264 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:33.755237    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:33.764324 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:33.764303    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:33.773329 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:33.773300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:34.548949 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:34.548918    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:34.559682 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:34.559651    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:34.569209 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:34.569184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:34.578821 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:34.578799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:35.099673 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:35.099635    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:35.366695 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:35.366597    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:35.387174 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:35.387144    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:35.403702 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:35.403675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:35.421177 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:35.421150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:36.212317 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:36.212284    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:36.223078 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:36.223025    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:36.232613 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:36.232588    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:36.242634 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:36.242596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:37.019195 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.019162    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:37.029967 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.029942    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:37.040197 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.040170    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:37.049966 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.049940    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:37.841926 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.841875    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:37.852869 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.852839    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:37.862288 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.862266    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:37.871760 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:37.871735    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:38.656973 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:38.656946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kserve-container/0.log"
Sep 09 20:38:38.667854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:38.667828    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/kube-rbac-proxy/0.log"
Sep 09 20:38:38.678392 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:38.678363    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/agent/0.log"
Sep 09 20:38:38.687656 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:38.687622    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-69444974bf-rrtdn_bc30d9b7-6e07-4726-8f48-e83456127b6e/storage-initializer/0.log"
Sep 09 20:38:40.100320 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:40.100273    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 20:38:40.105095 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:40.105046    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:38:40.105207 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:40.105193    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:40.105441 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:40.105418    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:40.105538 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:40.105527    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:42.133897 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.133869    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:42.200588 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200504    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config\") pod \"bc30d9b7-6e07-4726-8f48-e83456127b6e\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") "
Sep 09 20:38:42.200588 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200555    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location\") pod \"bc30d9b7-6e07-4726-8f48-e83456127b6e\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") "
Sep 09 20:38:42.200794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200616    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert\") pod \"bc30d9b7-6e07-4726-8f48-e83456127b6e\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") "
Sep 09 20:38:42.200794 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200686    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") pod \"bc30d9b7-6e07-4726-8f48-e83456127b6e\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") "
Sep 09 20:38:42.200903 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200881    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc30d9b7-6e07-4726-8f48-e83456127b6e" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:42.200957 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200925    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc30d9b7-6e07-4726-8f48-e83456127b6e" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:42.201017 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.200965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bc30d9b7-6e07-4726-8f48-e83456127b6e" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:42.202830 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.202807    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc30d9b7-6e07-4726-8f48-e83456127b6e" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:42.301210 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.301176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sjvr\" (UniqueName: \"kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr\") pod \"bc30d9b7-6e07-4726-8f48-e83456127b6e\" (UID: \"bc30d9b7-6e07-4726-8f48-e83456127b6e\") "
Sep 09 20:38:42.301393 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.301365    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc30d9b7-6e07-4726-8f48-e83456127b6e-kserve-provision-location\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:42.301393 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.301377    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-cabundle-cert\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:42.301393 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.301388    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc30d9b7-6e07-4726-8f48-e83456127b6e-proxy-tls\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:42.301498 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.301397    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:42.303250 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.303226    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr" (OuterVolumeSpecName: "kube-api-access-2sjvr") pod "bc30d9b7-6e07-4726-8f48-e83456127b6e" (UID: "bc30d9b7-6e07-4726-8f48-e83456127b6e"). InnerVolumeSpecName "kube-api-access-2sjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:42.402149 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.402111    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjvr\" (UniqueName: \"kubernetes.io/projected/bc30d9b7-6e07-4726-8f48-e83456127b6e-kube-api-access-2sjvr\") on node \"ip-10-0-143-149.ec2.internal\" DevicePath \"\""
Sep 09 20:38:42.704738 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.704701    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerID="41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3" exitCode=0
Sep 09 20:38:42.704920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.704779    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"
Sep 09 20:38:42.704920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.704783    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerDied","Data":"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"}
Sep 09 20:38:42.704920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.704878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn" event={"ID":"bc30d9b7-6e07-4726-8f48-e83456127b6e","Type":"ContainerDied","Data":"bab29165fd57ac1249bee3743897ced53131808b7f24fe6067a6906dbf381551"}
Sep 09 20:38:42.704920 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.704912    2564 scope.go:117] "RemoveContainer" containerID="41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"
Sep 09 20:38:42.714369 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.714349    2564 scope.go:117] "RemoveContainer" containerID="06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"
Sep 09 20:38:42.722785 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.722762    2564 scope.go:117] "RemoveContainer" containerID="26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"
Sep 09 20:38:42.727189 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.727161    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:38:42.731555 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.731526    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-69444974bf-rrtdn"]
Sep 09 20:38:42.732561 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.732540    2564 scope.go:117] "RemoveContainer" containerID="df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"
Sep 09 20:38:42.740483 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.740463    2564 scope.go:117] "RemoveContainer" containerID="41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"
Sep 09 20:38:42.740803 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:42.740782    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3\": container with ID starting with 41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3 not found: ID does not exist" containerID="41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"
Sep 09 20:38:42.740889 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.740813    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3"} err="failed to get container status \"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3\": rpc error: code = NotFound desc = could not find container \"41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3\": container with ID starting with 41d0464f37127e378e4edbd6029389d1a6c06f034ea58b159558c21fe3f517e3 not found: ID does not exist"
Sep 09 20:38:42.740889 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.740833    2564 scope.go:117] "RemoveContainer" containerID="06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"
Sep 09 20:38:42.741120 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:42.741099    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c\": container with ID starting with 06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c not found: ID does not exist" containerID="06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"
Sep 09 20:38:42.741172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.741120    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c"} err="failed to get container status \"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c\": rpc error: code = NotFound desc = could not find container \"06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c\": container with ID starting with 06b18ab861d92f3ef82b80115faec50aaf9f5102e3ebbe5e65845d9d7800e65c not found: ID does not exist"
Sep 09 20:38:42.741172 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.741132    2564 scope.go:117] "RemoveContainer" containerID="26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"
Sep 09 20:38:42.741408 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:42.741390    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e\": container with ID starting with 26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e not found: ID does not exist" containerID="26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"
Sep 09 20:38:42.741457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.741414    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e"} err="failed to get container status \"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e\": rpc error: code = NotFound desc = could not find container \"26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e\": container with ID starting with 26d1946823719da235e36d6a2189c8842b7d05b76b2a9967cc43c7b6da83874e not found: ID does not exist"
Sep 09 20:38:42.741457 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.741429    2564 scope.go:117] "RemoveContainer" containerID="df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"
Sep 09 20:38:42.741649 ip-10-0-143-149 kubenswrapper[2564]: E0909 20:38:42.741633    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380\": container with ID starting with df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380 not found: ID does not exist" containerID="df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"
Sep 09 20:38:42.741688 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:42.741652    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380"} err="failed to get container status \"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380\": rpc error: code = NotFound desc = could not find container \"df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380\": container with ID starting with df1a3818ac9f355a75fc4c436fe0225af5bc852a800fa4cadfad81bc38beb380 not found: ID does not exist"
Sep 09 20:38:43.119099 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.119048    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" path="/var/lib/kubelet/pods/bc30d9b7-6e07-4726-8f48-e83456127b6e/volumes"
Sep 09 20:38:43.989014 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.988979    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6t7hr/must-gather-qvbpx"]
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989350    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989362    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989379    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kserve-container"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989385    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kserve-container"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989392    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989397    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989403    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="storage-initializer"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989408    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="storage-initializer"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989419    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989424    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989430    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.989446 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989435    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.989854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989484    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kserve-container"
Sep 09 20:38:43.989854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989493    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.989854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989499    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="kserve-container"
Sep 09 20:38:43.989854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989505    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc30d9b7-6e07-4726-8f48-e83456127b6e" containerName="agent"
Sep 09 20:38:43.989854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.989511    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ef2777-6ecf-4055-9af9-81a08f1d685e" containerName="kube-rbac-proxy"
Sep 09 20:38:43.994149 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.994132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:43.996374 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.996353    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t7hr\"/\"openshift-service-ca.crt\""
Sep 09 20:38:43.996578 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.996565    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6t7hr\"/\"default-dockercfg-lgrms\""
Sep 09 20:38:43.996639 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:43.996613    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t7hr\"/\"kube-root-ca.crt\""
Sep 09 20:38:44.001581 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.001447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/must-gather-qvbpx"]
Sep 09 20:38:44.014323 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.014294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-must-gather-output\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.014497 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.014363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rh6\" (UniqueName: \"kubernetes.io/projected/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-kube-api-access-g4rh6\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.115023 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.114989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rh6\" (UniqueName: \"kubernetes.io/projected/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-kube-api-access-g4rh6\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.115263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.115151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-must-gather-output\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.115501 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.115481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-must-gather-output\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.122610 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.122581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rh6\" (UniqueName: \"kubernetes.io/projected/ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b-kube-api-access-g4rh6\") pod \"must-gather-qvbpx\" (UID: \"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b\") " pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.314770 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.314735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/must-gather-qvbpx"
Sep 09 20:38:44.583817 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.583669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/must-gather-qvbpx"]
Sep 09 20:38:44.586539 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:38:44.586511    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae597bdb_b3f5_4ede_9de7_f08b3a0d5d8b.slice/crio-d151e35bee4ad4aeb3a7d8220406184183a9e73df20ea3bd22a19fc5a3f9c4ed WatchSource:0}: Error finding container d151e35bee4ad4aeb3a7d8220406184183a9e73df20ea3bd22a19fc5a3f9c4ed: Status 404 returned error can't find the container with id d151e35bee4ad4aeb3a7d8220406184183a9e73df20ea3bd22a19fc5a3f9c4ed
Sep 09 20:38:44.588616 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.588597    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:38:44.717454 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:44.717408    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/must-gather-qvbpx" event={"ID":"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b","Type":"ContainerStarted","Data":"d151e35bee4ad4aeb3a7d8220406184183a9e73df20ea3bd22a19fc5a3f9c4ed"}
Sep 09 20:38:45.729071 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:45.728954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/must-gather-qvbpx" event={"ID":"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b","Type":"ContainerStarted","Data":"6c18500e213c52c47e5c105655e430e0520ea11e30db8cb94eab2f9925f6fab6"}
Sep 09 20:38:45.729071 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:45.729007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/must-gather-qvbpx" event={"ID":"ae597bdb-b3f5-4ede-9de7-f08b3a0d5d8b","Type":"ContainerStarted","Data":"ef02fa976d61d7c2ba5f124c59d726992b8ff6b0b0107b60c41b2cb9303d1e3f"}
Sep 09 20:38:45.746090 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:45.745857    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6t7hr/must-gather-qvbpx" podStartSLOduration=1.89297879 podStartE2EDuration="2.745842203s" totalImagesPullingTime="852.863413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:45.745198145 +0000 UTC m=+1665.119736241" watchObservedRunningTime="2026-09-09 20:38:45.745842203 +0000 UTC m=+1665.120380297"
Sep 09 20:38:46.869353 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:46.869315    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rn6b9_c0a9382c-c5ae-4a35-8e07-09ea6026a485/global-pull-secret-syncer/0.log"
Sep 09 20:38:46.930360 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:46.930332    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g5zrk_911f6be8-2fc6-40bb-9694-7e018bbb972a/konnectivity-agent/0.log"
Sep 09 20:38:47.000156 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:47.000117    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-149.ec2.internal_9005ed891fe314d0a7d8b24d4cb67be1/haproxy/0.log"
Sep 09 20:38:51.108259 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.108219    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-9zrpr_974ebb76-2a57-4c95-971c-6f6be973a3d9/cluster-monitoring-operator/0.log"
Sep 09 20:38:51.154957 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.154924    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-lbtd4_44aebd4a-7a3b-41b6-899f-ac5655cf7ec7/kube-state-metrics/0.log"
Sep 09 20:38:51.179894 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.179847    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-lbtd4_44aebd4a-7a3b-41b6-899f-ac5655cf7ec7/kube-rbac-proxy-main/0.log"
Sep 09 20:38:51.206859 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.206827    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-lbtd4_44aebd4a-7a3b-41b6-899f-ac5655cf7ec7/kube-rbac-proxy-self/0.log"
Sep 09 20:38:51.280290 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.280262    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-4rjzm_cbf01d2a-e87c-467a-8d0e-c60ce9736cec/monitoring-plugin/0.log"
Sep 09 20:38:51.469604 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.469481    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj7fv_c58100be-3a65-4ad2-86ee-78288a119029/node-exporter/0.log"
Sep 09 20:38:51.493829 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.493794    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj7fv_c58100be-3a65-4ad2-86ee-78288a119029/kube-rbac-proxy/0.log"
Sep 09 20:38:51.522070 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.520787    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj7fv_c58100be-3a65-4ad2-86ee-78288a119029/init-textfile/0.log"
Sep 09 20:38:51.551732 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.551695    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jhrzr_53f6ad1d-cdc5-424f-9685-880228aa6251/kube-rbac-proxy-main/0.log"
Sep 09 20:38:51.576312 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.576269    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jhrzr_53f6ad1d-cdc5-424f-9685-880228aa6251/kube-rbac-proxy-self/0.log"
Sep 09 20:38:51.600712 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.600678    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-jhrzr_53f6ad1d-cdc5-424f-9685-880228aa6251/openshift-state-metrics/0.log"
Sep 09 20:38:51.877010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.876975    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75db8b8f67-m2drk_6ad2ea8f-ea87-444a-9781-6cee9319ab07/telemeter-client/0.log"
Sep 09 20:38:51.898981 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.898945    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75db8b8f67-m2drk_6ad2ea8f-ea87-444a-9781-6cee9319ab07/reload/0.log"
Sep 09 20:38:51.925923 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:51.925890    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75db8b8f67-m2drk_6ad2ea8f-ea87-444a-9781-6cee9319ab07/kube-rbac-proxy/0.log"
Sep 09 20:38:53.420335 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:53.420294    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-s6c5t_fb8161c5-4de2-42be-b02e-c73ac0a5c015/networking-console-plugin/0.log"
Sep 09 20:38:53.956853 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:53.956822    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/2.log"
Sep 09 20:38:53.962941 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:53.962914    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-lmvxs_cdea4a8f-b525-4646-b6dd-39da4ba10271/console-operator/3.log"
Sep 09 20:38:54.359766 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:54.359737    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f79fb54c7-zmcxj_b5100a4b-1731-4481-aef5-f9dfa2df9831/console/0.log"
Sep 09 20:38:54.897540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:54.897512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-49tn2_032fd8cb-9219-4537-944d-9352add4873f/volume-data-source-validator/0.log"
Sep 09 20:38:55.853325 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.853287    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"]
Sep 09 20:38:55.860781 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.860754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.872351 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.872316    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"]
Sep 09 20:38:55.912214 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.912181    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kpld5_8544c883-e15c-40ee-90db-19caf5a8b6ea/dns/0.log"
Sep 09 20:38:55.923121 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.923051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxthw\" (UniqueName: \"kubernetes.io/projected/c1692e24-c69d-4af1-8f26-668a7e632427-kube-api-access-hxthw\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.923297 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.923179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-proc\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.923297 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.923224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-lib-modules\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.923391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.923297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-podres\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.923391 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.923376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-sys\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:55.932862 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.932837    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kpld5_8544c883-e15c-40ee-90db-19caf5a8b6ea/kube-rbac-proxy/0.log"
Sep 09 20:38:55.998166 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:55.998140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtncg_6a2a57af-fe8b-420d-8083-d08bcce8dcaa/dns-node-resolver/0.log"
Sep 09 20:38:56.025094 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.025027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-sys\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.025448 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.025419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-sys\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.025610 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.025590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxthw\" (UniqueName: \"kubernetes.io/projected/c1692e24-c69d-4af1-8f26-668a7e632427-kube-api-access-hxthw\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026010 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.025980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-proc\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026132 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.026034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-lib-modules\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026132 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.026091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-podres\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026263 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.026245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-podres\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026375 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.026357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-lib-modules\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.026443 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.026357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1692e24-c69d-4af1-8f26-668a7e632427-proc\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.034555 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.034487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxthw\" (UniqueName: \"kubernetes.io/projected/c1692e24-c69d-4af1-8f26-668a7e632427-kube-api-access-hxthw\") pod \"perf-node-gather-daemonset-qb229\" (UID: \"c1692e24-c69d-4af1-8f26-668a7e632427\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.181927 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.181835    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.472950 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.472859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"]
Sep 09 20:38:56.477019 ip-10-0-143-149 kubenswrapper[2564]: W0909 20:38:56.476990    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1692e24_c69d_4af1_8f26_668a7e632427.slice/crio-87622390657cacff29ef68603f0998c43eb3f02449c3812235658b2e7fe55e1b WatchSource:0}: Error finding container 87622390657cacff29ef68603f0998c43eb3f02449c3812235658b2e7fe55e1b: Status 404 returned error can't find the container with id 87622390657cacff29ef68603f0998c43eb3f02449c3812235658b2e7fe55e1b
Sep 09 20:38:56.537705 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.537675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vsxdf_a9ba811a-5b54-43e3-8324-0925702101a0/node-ca/0.log"
Sep 09 20:38:56.809820 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.809769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229" event={"ID":"c1692e24-c69d-4af1-8f26-668a7e632427","Type":"ContainerStarted","Data":"9f61eddaede9cc6debd26e16ce08766f94d2d57683de8b852b8610d109d5df1b"}
Sep 09 20:38:56.810027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.809848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229" event={"ID":"c1692e24-c69d-4af1-8f26-668a7e632427","Type":"ContainerStarted","Data":"87622390657cacff29ef68603f0998c43eb3f02449c3812235658b2e7fe55e1b"}
Sep 09 20:38:56.810027 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.809881    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:38:56.828295 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:56.828238    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229" podStartSLOduration=1.828221471 podStartE2EDuration="1.828221471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:56.826134589 +0000 UTC m=+1676.200672685" watchObservedRunningTime="2026-09-09 20:38:56.828221471 +0000 UTC m=+1676.202759564"
Sep 09 20:38:57.297889 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:57.297863    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-87c784c5b-k6grx_5f74bf6b-25f7-439d-98c7-1b79b2e89c82/router/0.log"
Sep 09 20:38:57.644473 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:57.644392    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mbn2f_b713f385-521f-48a1-8d5e-2fc44e380c56/serve-healthcheck-canary/0.log"
Sep 09 20:38:57.975937 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:57.975857    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nh9z8_aba3eb39-9f90-4766-be49-f459acbf5379/insights-operator/1.log"
Sep 09 20:38:57.976140 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:57.976102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nh9z8_aba3eb39-9f90-4766-be49-f459acbf5379/insights-operator/0.log"
Sep 09 20:38:58.052555 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:58.052527    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nw29n_c7d1bd14-c0ae-4f9f-b680-f63734dfce35/kube-rbac-proxy/0.log"
Sep 09 20:38:58.070495 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:58.070467    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nw29n_c7d1bd14-c0ae-4f9f-b680-f63734dfce35/exporter/0.log"
Sep 09 20:38:58.088014 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:38:58.087987    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nw29n_c7d1bd14-c0ae-4f9f-b680-f63734dfce35/extractor/0.log"
Sep 09 20:39:00.416525 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.416479    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-55b9566966-qzg97_fab0065b-d50d-40ea-b1c8-b4fc7f0545a0/manager/0.log"
Sep 09 20:39:00.467252 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.467223    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-hpswd_48cd2969-5c95-459a-8700-fc627f7c6ddc/server/0.log"
Sep 09 20:39:00.599296 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.599251    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-25f7c_f9b2c9a9-a235-4f10-b68c-c84b6633d331/manager/0.log"
Sep 09 20:39:00.629540 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.629512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-87kfr_7c530915-b152-4042-ab5e-cd2d735b7fb8/s3-init/0.log"
Sep 09 20:39:00.643854 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.643826    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-87kfr_7c530915-b152-4042-ab5e-cd2d735b7fb8/download-hf-model/0.log"
Sep 09 20:39:00.666710 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:00.666625    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tj96n_4d2341f2-dcda-4d86-88dd-40d44b7a3a22/seaweedfs/0.log"
Sep 09 20:39:02.826728 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:02.826698    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-qb229"
Sep 09 20:39:04.874818 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:04.874772    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dcg2g_8e5dfd22-9ef7-488b-9819-5eb4fbba4035/kube-storage-version-migrator-operator/1.log"
Sep 09 20:39:04.876051 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:04.876015    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dcg2g_8e5dfd22-9ef7-488b-9819-5eb4fbba4035/kube-storage-version-migrator-operator/0.log"
Sep 09 20:39:05.893484 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.893457    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:39:05.914518 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.914458    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/egress-router-binary-copy/0.log"
Sep 09 20:39:05.932729 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.932700    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/cni-plugins/0.log"
Sep 09 20:39:05.954354 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.954323    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/bond-cni-plugin/0.log"
Sep 09 20:39:05.973804 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.973774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/routeoverride-cni/0.log"
Sep 09 20:39:05.992530 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:05.992507    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/whereabouts-cni-bincopy/0.log"
Sep 09 20:39:06.011661 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:06.011575    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-74l2q_d40242c9-05a1-4c5e-aa11-936ace80d2dd/whereabouts-cni/0.log"
Sep 09 20:39:06.315437 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:06.315396    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fqlhz_a6b3ecd2-3ca5-4fbb-89ff-5bb610b05491/kube-multus/0.log"
Sep 09 20:39:06.421040 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:06.421008    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l82g9_e2feec63-e0b9-4f53-ac1c-b4e4c876c9df/network-metrics-daemon/0.log"
Sep 09 20:39:06.443533 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:06.443506    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l82g9_e2feec63-e0b9-4f53-ac1c-b4e4c876c9df/kube-rbac-proxy/0.log"
Sep 09 20:39:07.474851 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.474824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-controller/0.log"
Sep 09 20:39:07.490463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.490422    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/1.log"
Sep 09 20:39:07.490463 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.490440    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovn-acl-logging/0.log"
Sep 09 20:39:07.506857 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.506828    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/kube-rbac-proxy-node/0.log"
Sep 09 20:39:07.525352 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.525323    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:39:07.542605 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.542578    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/northd/0.log"
Sep 09 20:39:07.561614 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.561538    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/nbdb/0.log"
Sep 09 20:39:07.581433 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.581395    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/sbdb/0.log"
Sep 09 20:39:07.626049 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:07.626015    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q8td4_ed289824-ea7f-446b-b9f7-b51a6b31a34d/ovnkube-controller/0.log"
Sep 09 20:39:08.780365 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:08.780333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-mnn2h_f749172f-1796-42aa-9d8e-29c96523be1e/check-endpoints/0.log"
Sep 09 20:39:08.845218 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:08.845189    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mtz8b_18ab09d5-8034-465e-a2fd-f53a5ed05581/network-check-target-container/0.log"
Sep 09 20:39:09.753224 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:09.753194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vfjzd_99fec714-3a7f-47a4-8b2b-743bc4d1b60a/iptables-alerter/0.log"
Sep 09 20:39:10.352139 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:10.352108    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fdz7p_c4bda37d-7d2d-441b-9e03-57bb934a62e1/tuned/0.log"
Sep 09 20:39:11.982568 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:11.982534    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-mgs55_d70fa5fc-4c59-48d9-895c-591e860629f3/cluster-samples-operator/0.log"
Sep 09 20:39:11.997440 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:11.997410    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-mgs55_d70fa5fc-4c59-48d9-895c-591e860629f3/cluster-samples-operator-watch/0.log"
Sep 09 20:39:12.937347 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:12.937317    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-x8nlz_3affe817-470e-426a-aeae-1e75a35602dc/service-ca-operator/1.log"
Sep 09 20:39:12.938433 ip-10-0-143-149 kubenswrapper[2564]: I0909 20:39:12.938408    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-x8nlz_3affe817-470e-426a-aeae-1e75a35602dc/service-ca-operator/0.log"