Jul 28 12:35:10.055195 ip-10-0-134-201 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:35:10.430377 ip-10-0-134-201 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:10.430377 ip-10-0-134-201 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:35:10.430377 ip-10-0-134-201 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:10.430377 ip-10-0-134-201 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:35:10.430377 ip-10-0-134-201 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:10.431569 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.431444    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:35:10.439028 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439004    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.439028 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439027    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.439028 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439032    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439037    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439042    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439045    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439049    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439053    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439056    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439059    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439062    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439065    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439068    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439071    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439074    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439077    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439080    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439083    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439086    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439089    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439092    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.439176 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439094    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439097    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439100    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439102    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439105    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439108    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439111    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439123    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439126    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439129    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439132    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439135    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439138    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439142    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439145    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439148    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439151    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439155    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439158    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439161    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.439673 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439164    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439166    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439169    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439172    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439175    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439178    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439182    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439184    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439187    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439190    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439193    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439197    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439200    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439202    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439205    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439208    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439213    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439218    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439221    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439224    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.440233 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439227    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439232    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439236    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439239    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439242    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439246    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439249    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439252    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439255    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439258    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439261    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439263    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439266    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439269    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439272    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439274    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439277    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439280    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439283    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439285    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.440805 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439288    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439291    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439294    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439297    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439300    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439303    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439306    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439309    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439311    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439314    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439317    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439320    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439323    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439326    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439328    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439331    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439334    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.439337    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440194    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.441358 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440203    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440207    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440211    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440214    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440218    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440221    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440224    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440228    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440231    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440235    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440238    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440242    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440244    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440248    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440251    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440254    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440257    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440260    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440263    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440266    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.441878 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440269    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440273    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440276    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440279    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440281    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440284    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440287    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440290    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440293    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440296    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440299    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440302    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440305    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440307    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440310    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440313    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440316    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440319    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440321    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440324    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.442421 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440327    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440329    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440332    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440335    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440338    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440341    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440343    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440346    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440349    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440352    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440355    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440358    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440362    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440365    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440368    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440371    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440373    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440376    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440379    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440382    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.442962 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440385    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440389    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440394    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440397    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440401    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440405    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440409    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440412    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440415    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440418    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440420    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440424    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440428    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440431    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440435    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440439    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440442    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440446    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440449    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.443495 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440452    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440455    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440457    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440481    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440485    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440489    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440493    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440497    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440501    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440504    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440506    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440509    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440512    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440515    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440517    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440520    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440523    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440526    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.440529    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.444026 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441023    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441030    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441033    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441036    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441039    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441043    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441045    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441049    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441051    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441054    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441057    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441060    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441063    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441065    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441068    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441071    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441076    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441080    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441085    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.444549 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441089    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441092    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441095    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441098    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441102    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441105    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441109    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441112    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441114    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441117    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441120    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441123    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441126    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441129    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441131    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441134    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441138    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441142    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441145    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.445072 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441148    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441151    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441153    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441157    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441159    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441162    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441165    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441167    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441170    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441173    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441175    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441179    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441182    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441185    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441188    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441191    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441193    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441196    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441199    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441202    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.445578 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441204    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441207    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441210    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441212    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441215    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441218    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441220    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441223    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441226    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441229    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441231    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441234    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441237    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441239    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441242    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441245    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441247    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441250    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441253    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.446186 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441255    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441258    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441261    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441264    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441266    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441272    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441275    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441278    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441280    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441283    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441286    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441290    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441292    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441295    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441298    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441300    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441303    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441308    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441310    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441314    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.446685 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441316    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.441319    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441400    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441409    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441416    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441424    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441428    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441432    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441436    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441441    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441444    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441447    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441451    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441455    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441458    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441461    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441465    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441469    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441473    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441477    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441481    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441485    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441488    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441492    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:35:10.447243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441495    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441500    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441503    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441506    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441510    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441513    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441517    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441520    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441523    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441527    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441532    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441535    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441539    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441543    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441547    2560 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441551    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441556    2560 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441559    2560 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441563    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441566    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441569    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441574    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441577    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441581    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441584    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:35:10.447881 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441589    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441602    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441606    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441610    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441613    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441616    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441620    2560 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441624    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441627    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441631    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441634    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441638    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441641    2560 flags.go:64] FLAG: --help="false"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441644    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441647    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441651    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441654    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441657    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441661    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441664    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441670    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441673    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441676    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:35:10.448553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441679    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441683    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441686    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441689    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441692    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441696    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441699    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441702    2560 flags.go:64] FLAG: --lock-file=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441705    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441710    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441713    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441719    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441722    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441726    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441729    2560 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441733    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441736    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441740    2560 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441743    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441753    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441756    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441761    2560 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441764    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441767    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441770    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:35:10.449250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441773    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441777    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441780    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441783    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441795    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441799    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441802    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441805    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441808    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441815    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441818    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441821    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441825    2560 flags.go:64] FLAG: --port="10250"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441828    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441831    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01b402cc7edb39c04"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441835    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441838    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441843    2560 flags.go:64] FLAG: --register-node="true"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441846    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441850    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441854    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441857    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441861    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441865    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441868    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:35:10.450143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441871    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441874    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441878    2560 flags.go:64] FLAG: --runonce="false"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441881    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441884    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441888    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441891    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441894    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441898    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441901    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441904    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441909    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441912    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441915    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441918    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441922    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441925    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441928    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441934    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441953    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441958    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441965    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441968    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441971    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441975    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:35:10.450823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441978    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441982    2560 flags.go:64] FLAG: --v="2"
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441986    2560 flags.go:64] FLAG: --version="false"
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441991    2560 flags.go:64] FLAG: --vmodule=""
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441995    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.441999    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442124    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442128    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442132    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442136    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442140    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442144    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442147    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442150    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442154    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442157    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442160    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442163    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442167    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442170    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442173    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.451517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442176    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442179    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442182    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442184    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442187    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442190    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442193    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442195    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442198    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442201    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442205    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442208    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442211    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442214    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442216    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442221    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442226    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442229    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442232    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.452082 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442236    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442239    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442242    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442245    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442248    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442251    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442254    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442257    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442260    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442263    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442267    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442270    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442273    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442276    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442278    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442281    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442284    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442287    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442290    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442292    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.452581 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442295    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442299    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442303    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442308    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442311    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442314    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442317    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442319    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442322    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442326    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442329    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442331    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442334    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442337    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442340    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442343    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442346    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442349    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442352    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.453140 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442355    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442357    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442361    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442364    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442367    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442370    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442373    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442376    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442379    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442381    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442384    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442387    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442390    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442393    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442395    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442398    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442402    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442405    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442408    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442411    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.453641 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442413    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442416    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442419    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442422    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442426    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.442429    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.443262    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.450438    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.450455    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.450477    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.450487    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450547    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450552    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450555    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450558    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.454247 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450561    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450566    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450570    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450573    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450576    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450580    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450583    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450587    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450589    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450592    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450595    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450598    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450601    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450604    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450606    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450609    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450612    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450615    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450617    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.454658 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450620    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450623    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450626    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450628    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450631    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450634    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450638    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450642    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450644    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450647    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450650    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450653    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450655    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450658    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450661    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450664    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450667    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450669    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450672    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450675    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.455184 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450678    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450681    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450684    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450687    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450689    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450692    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450695    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450697    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450700    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450703    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450706    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450709    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450712    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450715    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450718    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450720    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450723    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450726    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450730    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450733    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.455694 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450736    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450739    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450742    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450744    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450747    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450750    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450753    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450756    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450759    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450762    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450764    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450767    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450770    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450773    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450777    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450781    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450785    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450788    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450790    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450793    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.456264 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450796    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450799    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450802    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450805    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450807    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450810    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450813    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450817    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450820    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450824    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450828    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450831    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450835    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450838    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450841    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.456787 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450843    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.450849    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.450994    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451001    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451005    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451008    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451011    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451013    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451016    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451019    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451022    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451025    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451028    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451031    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451034    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:10.457211 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451037    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451040    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451043    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451045    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451048    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451051    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451053    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451056    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451059    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451061    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451064    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451068    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451072    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451074    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451077    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451080    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451083    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451086    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451089    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451091    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:10.457621 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451094    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451097    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451099    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451102    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451104    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451107    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451110    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451113    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451115    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451118    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451121    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451123    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451126    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451129    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451131    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451134    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451137    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451139    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451142    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451145    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:10.458167 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451147    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451150    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451153    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451155    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451159    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451162    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451165    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451168    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451171    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451174    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451177    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451180    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451182    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451185    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451188    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451192    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451195    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451198    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451201    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:10.458692 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451205    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451210    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451213    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451216    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451219    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451222    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451224    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451227    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451230    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451233    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451236    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451238    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451241    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451243    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451246    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451249    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451252    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451254    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451257    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451260    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:10.459207 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451262    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451265    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451268    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451271    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451273    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451277    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:10.451281    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.451286    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.451440    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.453563    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.454933    2560 server.go:1019] "Starting client certificate rotation"
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.455052    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.455092    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:35:10.459730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.457423    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:35:10.460134 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.458668    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:35:10.472629 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.472606    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:35:10.474760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.474744    2560 log.go:25] "Validated CRI v1 image API"
Jul 28 12:35:10.475620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.475601    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:35:10.478295 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.478269    2560 fs.go:135] Filesystem UUIDs: map[4e397f95-dd39-4e73-8f66-5248813c9463:/dev/nvme0n1p4 595c5c52-ec67-410a-8e0b-b7b9f5f13757:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 12:35:10.478383 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.478290    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:35:10.483781 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.483652    2560 manager.go:216] Machine: {Timestamp:2026-07-28 12:35:10.482142057 +0000 UTC m=+0.316887236 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100923 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20051f6336d442e3e3095e75bf6958 SystemUUID:ec20051f-6336-d442-e3e3-095e75bf6958 BootID:42029cc7-2fad-410a-92b0-ab40bb99e1fe Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:4c:f3:7d:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:4c:f3:7d:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:c1:1f:e3:42:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:35:10.483781 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.483772    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:35:10.483905 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.483864    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:35:10.485933 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.485907    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:35:10.486109 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.485951    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:35:10.487583 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.487563    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:35:10.487632 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.487587    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:35:10.487632 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.487596    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:35:10.488357 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.488345    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:35:10.489706 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.489693    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:35:10.491419 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.491403    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:35:10.492641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492628    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:35:10.492771 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492761    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:35:10.492810 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492796    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:35:10.492810 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492806    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:35:10.492873 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492826    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:35:10.492873 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492838    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:35:10.492873 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.492858    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:35:10.493242 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.493230    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:35:10.493296 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.493249    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:35:10.495612 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.495595    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:35:10.497573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.497559    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:35:10.497639 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.497591    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:35:10.499115 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499103    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499119    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499127    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499133    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499142    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499148    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499156    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499162    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499169    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:35:10.499176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499174    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:35:10.499437 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499202    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:35:10.499437 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.499208    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:35:10.500154 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.500143    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:35:10.500154 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.500154    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:35:10.503574 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.503557    2560 server.go:1268] "Started kubelet"
Jul 28 12:35:10.503680 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.503652    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:35:10.504228 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.503971    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:35:10.504228 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.503972    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:35:10.504228 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.504140    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:35:10.504228 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.504114    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:35:10.504228 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.504190    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:35:10.504536 ip-10-0-134-201 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:35:10.509008 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.508982    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:35:10.510541 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.510512    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:35:10.511103 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.511075    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:35:10.513631 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.512667    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-201.ec2.internal.18c6743daaa19d68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-201.ec2.internal,UID:ip-10-0-134-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-201.ec2.internal,},FirstTimestamp:2026-07-28 12:35:10.503525736 +0000 UTC m=+0.338271128,LastTimestamp:2026-07-28 12:35:10.503525736 +0000 UTC m=+0.338271128,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-201.ec2.internal,}"
Jul 28 12:35:10.515855 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.515839    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:35:10.515920 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.515850    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:35:10.516105 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.516087    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.516697 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516586    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:35:10.516697 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516661    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:35:10.516697 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516675    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:35:10.516872 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516778    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:35:10.516872 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516789    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:35:10.517003 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516932    2560 factory.go:55] Registering systemd factory
Jul 28 12:35:10.517003 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.516948    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:35:10.517212 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517198    2560 factory.go:153] Registering CRI-O factory
Jul 28 12:35:10.517261 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517216    2560 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:35:10.517293 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517264    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:35:10.517293 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517283    2560 factory.go:103] Registering Raw factory
Jul 28 12:35:10.517352 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517295    2560 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:35:10.517861 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.517840    2560 manager.go:349] Starting recovery of all containers
Jul 28 12:35:10.519337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.519309    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lql8d"
Jul 28 12:35:10.522013 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.521977    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:35:10.522176 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.522147    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:35:10.526584 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.526558    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lql8d"
Jul 28 12:35:10.529208 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.529191    2560 manager.go:354] Recovery completed
Jul 28 12:35:10.534330 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.534197    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.536378 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.536358    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.536450 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.536401    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.536450 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.536414    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.537035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.537017    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:35:10.537035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.537035    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:35:10.537162 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.537037    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.537162 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.537055    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:35:10.538191 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.538084    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-201.ec2.internal.18c6743dac96ec92  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-201.ec2.internal,UID:ip-10-0-134-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-201.ec2.internal,},FirstTimestamp:2026-07-28 12:35:10.536379538 +0000 UTC m=+0.371124713,LastTimestamp:2026-07-28 12:35:10.536379538 +0000 UTC m=+0.371124713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-201.ec2.internal,}"
Jul 28 12:35:10.538311 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538288    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.538311 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538310    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.538391 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538323    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.538684 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538671    2560 policy_none.go:49] "None policy: Start"
Jul 28 12:35:10.538735 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538690    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:35:10.538735 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.538702    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:35:10.539894 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.539881    2560 policy_none.go:47] "Start"
Jul 28 12:35:10.575616 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.575592    2560 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:35:10.575757 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.575632    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:35:10.575757 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.575647    2560 server.go:87] "Starting device plugin registration server"
Jul 28 12:35:10.576000 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.575981    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:35:10.576117 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.575999    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:35:10.576117 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.576082    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:35:10.576352 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.576164    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:35:10.576352 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.576174    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:35:10.576858 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.576833    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:35:10.576958 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.576922    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.654242 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.654197    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:35:10.655570 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.655553    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:35:10.655674 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.655599    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:35:10.655674 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.655630    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:35:10.655772 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.655678    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:35:10.658349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.658325    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:10.676443 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.676400    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.677546 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.677529    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.677624 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.677563    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.677624 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.677578    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.677624 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.677604    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.686290 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.686233    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.686290 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.686263    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-201.ec2.internal\": node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.702630 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.702604    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.757052 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.757014    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"]
Jul 28 12:35:10.757184 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.757099    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.758209 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.758193    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.758272 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.758225    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.758272 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.758235    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.759924 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.759910    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.760047 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760032    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.760107 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760064    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.760921 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760904    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.761121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760930    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.761121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760908    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.761121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760968    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.761121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760984    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.761121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.760941    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.762287 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.762270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.762342 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.762307    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:10.763139 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.763110    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:10.763139 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.763136    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:10.763294 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.763154    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:10.784821 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.784792    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-201.ec2.internal\" not found" node="ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.789670 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.789651    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-201.ec2.internal\" not found" node="ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.802786 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.802766    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.817728 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.817707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ce3bf83fccda127909e8840dbc1840-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"61ce3bf83fccda127909e8840dbc1840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.817815 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.817731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.817815 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.817747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.903620 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:10.903572    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:10.917939 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.917913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ce3bf83fccda127909e8840dbc1840-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"61ce3bf83fccda127909e8840dbc1840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.918028 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.917944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.918028 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.917971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.918109 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.918036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ce3bf83fccda127909e8840dbc1840-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"61ce3bf83fccda127909e8840dbc1840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.918109 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.918088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:10.918184 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:10.918125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d10ec0ce97f3475d94c29e5baee4c519-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"d10ec0ce97f3475d94c29e5baee4c519\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:11.003822 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.003743    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:11.090135 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.090104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:11.094228 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.094199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:11.104334 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.104308    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:11.204994 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.204927    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:11.309832 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.309752    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:11.410926 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.410893    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found"
Jul 28 12:35:11.439764 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.439731    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:11.454954 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.454926    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:35:11.455128 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.455106    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:35:11.455128 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.455112    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:35:11.493630 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.493598    2560 apiserver.go:52] "Watching apiserver"
Jul 28 12:35:11.513418 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.513392    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:35:11.514993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.514962    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vs7wx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d","openshift-cluster-node-tuning-operator/tuned-sx2xr","openshift-dns/node-resolver-crz4p","openshift-image-registry/node-ca-8ncht","openshift-multus/multus-7qmmz","openshift-multus/multus-additional-cni-plugins-bvm2g","openshift-network-diagnostics/network-check-target-ksczq","kube-system/global-pull-secret-syncer-llr2w","openshift-multus/network-metrics-daemon-86zqw","openshift-network-operator/iptables-alerter-rpf64","openshift-ovn-kubernetes/ovnkube-node-9jw6s"]
Jul 28 12:35:11.516240 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.516216    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:35:11.516240 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.516236    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:11.516413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.516384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.517781 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.517752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.519153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.519133    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:35:11.519254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.519153    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-klt45\""
Jul 28 12:35:11.519254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.519160    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:35:11.519363 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.519310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.520384 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.520360    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hm8wr\""
Jul 28 12:35:11.520487 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.520399    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.520487 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.520438    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.520487 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.520400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:35:11.520995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.520979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-socket-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-registration-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-device-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-agent-certs\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-konnectivity-ca\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.521637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-sys-fs\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.522060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.521604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjms\" (UniqueName: \"kubernetes.io/projected/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kube-api-access-hwjms\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.522249 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.522228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsxjm\""
Jul 28 12:35:11.522303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.522272    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.522610 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.522587    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.523470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.523068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.523818 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.523794    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.524185 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.524162    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.524316 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.524267    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gpzk4\""
Jul 28 12:35:11.524603 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.524582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.525666 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.525647    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.525837 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.525821    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jk5jd\""
Jul 28 12:35:11.525912 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.525844    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:35:11.526267 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.526252    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.526364 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.526348    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.527200 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jtwkl\""
Jul 28 12:35:11.527445 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527429    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:35:11.527721 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527702    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.527721 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527718    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:35:11.527854 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527804    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"
Jul 28 12:35:11.527854 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.527840    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:11.527994 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.527939    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:11.528686 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.528626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.528686 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.528663    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:35:11.528892 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.528724    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:35:11.529024 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.528968    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:35:11.529024 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.528993    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xxws\""
Jul 28 12:35:11.529438 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.529420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.529539 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.529478    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:11.530382 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.530239    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:35:11.530474 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.530394    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:30:10 +0000 UTC" deadline="2028-03-12 22:36:19.176435266 +0000 UTC"
Jul 28 12:35:11.530474 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.530422    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14242h1m7.646015271s"
Jul 28 12:35:11.530828 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.530810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.530925 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.530903    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:11.532034 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.532016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.533513 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.533494    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"]
Jul 28 12:35:11.533629 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.533596    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.534467 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.534449    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pfwfj\""
Jul 28 12:35:11.534566 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.534512    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:35:11.534750 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.534731    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.534836 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.534756    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.536025 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.536009    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:35:11.536538 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.536521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:35:11.536632 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.536560    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:35:11.536796 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.536783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:35:11.537896 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.537879    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:35:11.537987 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.537920    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7b9w6\""
Jul 28 12:35:11.537987 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.537932    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:35:11.542267 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.542251    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:35:11.543195 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.543174    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"]
Jul 28 12:35:11.552114 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.552094    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6988"
Jul 28 12:35:11.560328 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.560265    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6988"
Jul 28 12:35:11.617537 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.617511    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:35:11.618571 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.618553    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:11.621890 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.621868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.622015 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.621901    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-socket-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.622015 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.621921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-var-lib-kubelet\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622015 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.621938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-host\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622015 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-cni-binary-copy\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-kubelet\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fsc\" (UniqueName: \"kubernetes.io/projected/8b3873af-4061-4796-ab9a-5c9d44bc5675-kube-api-access-b9fsc\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-socket-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-conf\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldf4\" (UniqueName: \"kubernetes.io/projected/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-kube-api-access-6ldf4\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-netns\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-node-log\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-registration-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysconfig\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-registration-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-kubernetes\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-system-cni-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5m8m\" (UniqueName: \"kubernetes.io/projected/3c2cfa72-ee71-41b3-9313-618a23234d37-kube-api-access-d5m8m\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-bin\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-lib-modules\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.622576 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-conf-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-etc-kubernetes\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdpl\" (UniqueName: \"kubernetes.io/projected/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-kube-api-access-bcdpl\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-system-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3c2cfa72-ee71-41b3-9313-618a23234d37-serviceca\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-kubelet-config\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b333236c-930f-4ac0-b950-b02e8e7374a9-host-slash\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjms\" (UniqueName: \"kubernetes.io/projected/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kube-api-access-hwjms\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvj7\" (UniqueName: \"kubernetes.io/projected/76047ac2-1c2f-4d1b-be99-9173a606cbef-kube-api-access-7hvj7\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-systemd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-tmp-dir\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-konnectivity-ca\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.622997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-tuned\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-socket-dir-parent\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2cfa72-ee71-41b3-9313-618a23234d37-host\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.623188 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-var-lib-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-etc-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-cnibin\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b333236c-930f-4ac0-b950-b02e8e7374a9-iptables-alerter-script\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-script-lib\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-systemd\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdlhs\" (UniqueName: \"kubernetes.io/projected/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-kube-api-access-fdlhs\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-systemd-units\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-sys\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-k8s-cni-cncf-io\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-netns\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-ovn\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-agent-certs\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-tmp\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-konnectivity-ca\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-os-release\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-hostroot\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-multus-certs\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lzx\" (UniqueName: \"kubernetes.io/projected/b333236c-930f-4ac0-b950-b02e8e7374a9-kube-api-access-q5lzx\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-slash\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.623982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-device-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-modprobe-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624035    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-bin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.624800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-device-dir\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-dbus\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/291e2747-1e3c-46df-bd4e-91631ff384f1-kube-api-access-hznjh\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-sys-fs\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-cnibin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-multus\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-log-socket\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-sys-fs\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-hosts-file\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-os-release\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-kubelet\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-netd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-config\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-env-overrides\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.625602 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-run\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.626422 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.624553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-daemon-config\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.627868 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.627845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5bbcca4-dfd3-4609-b22f-88d9d002ae80-agent-certs\") pod \"konnectivity-agent-vs7wx\" (UID: \"f5bbcca4-dfd3-4609-b22f-88d9d002ae80\") " pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.637146 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.637115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjms\" (UniqueName: \"kubernetes.io/projected/16365c9c-4da8-49fb-80d1-5974d4a7d3b3-kube-api-access-hwjms\") pod \"aws-ebs-csi-driver-node-xnz6d\" (UID: \"16365c9c-4da8-49fb-80d1-5974d4a7d3b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.691951 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.691911    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ce3bf83fccda127909e8840dbc1840.slice/crio-5c746ac3e59102e25abcccb1308824e3a96e5308e352f5010d2ae0418e1d085c WatchSource:0}: Error finding container 5c746ac3e59102e25abcccb1308824e3a96e5308e352f5010d2ae0418e1d085c: Status 404 returned error can't find the container with id 5c746ac3e59102e25abcccb1308824e3a96e5308e352f5010d2ae0418e1d085c
Jul 28 12:35:11.693153 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.693115    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10ec0ce97f3475d94c29e5baee4c519.slice/crio-1e9aa54c74466b277ab9dc9761dec7407bad6085b436efce5a60aa582fcd3206 WatchSource:0}: Error finding container 1e9aa54c74466b277ab9dc9761dec7407bad6085b436efce5a60aa582fcd3206: Status 404 returned error can't find the container with id 1e9aa54c74466b277ab9dc9761dec7407bad6085b436efce5a60aa582fcd3206
Jul 28 12:35:11.698831 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.698814    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:35:11.725060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.725060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-var-lib-kubelet\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-host\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-cni-binary-copy\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-var-lib-kubelet\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-host\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-kubelet\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fsc\" (UniqueName: \"kubernetes.io/projected/8b3873af-4061-4796-ab9a-5c9d44bc5675-kube-api-access-b9fsc\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-conf\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldf4\" (UniqueName: \"kubernetes.io/projected/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-kube-api-access-6ldf4\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-kubelet\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-netns\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-netns\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725465    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-conf\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-node-log\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysconfig\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-kubernetes\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-node-log\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-system-cni-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5m8m\" (UniqueName: \"kubernetes.io/projected/3c2cfa72-ee71-41b3-9313-618a23234d37-kube-api-access-d5m8m\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysconfig\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-bin\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.725677 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-system-cni-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-kubernetes\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-bin\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-lib-modules\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-conf-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-etc-kubernetes\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdpl\" (UniqueName: \"kubernetes.io/projected/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-kube-api-access-bcdpl\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.726313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-cni-binary-copy\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.726754 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.725946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-system-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.726754 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-lib-modules\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.726754 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-conf-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.726904 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726797    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3c2cfa72-ee71-41b3-9313-618a23234d37-serviceca\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.726904 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-kubelet-config\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.727034 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-etc-kubernetes\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.727034 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b333236c-930f-4ac0-b950-b02e8e7374a9-host-slash\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.727137 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvj7\" (UniqueName: \"kubernetes.io/projected/76047ac2-1c2f-4d1b-be99-9173a606cbef-kube-api-access-7hvj7\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.727137 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-kubelet-config\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.727137 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.726850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-system-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.727137 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b333236c-930f-4ac0-b950-b02e8e7374a9-host-slash\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-systemd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-tmp-dir\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-tuned\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.727337 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-systemd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-socket-dir-parent\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3c2cfa72-ee71-41b3-9313-618a23234d37-serviceca\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2cfa72-ee71-41b3-9313-618a23234d37-host\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.727559    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-var-lib-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.727644    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:12.227623069 +0000 UTC m=+2.062368245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.727660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2cfa72-ee71-41b3-9313-618a23234d37-host\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-var-lib-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-etc-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-etc-openvswitch\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-cnibin\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-sysctl-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.727912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b333236c-930f-4ac0-b950-b02e8e7374a9-iptables-alerter-script\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-script-lib\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-systemd\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdlhs\" (UniqueName: \"kubernetes.io/projected/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-kube-api-access-fdlhs\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-cnibin\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-socket-dir-parent\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-systemd-units\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-sys\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-tmp-dir\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-k8s-cni-cncf-io\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.728547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-netns\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-ovn\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-tmp\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-os-release\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-hostroot\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.728922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-multus-certs\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lzx\" (UniqueName: \"kubernetes.io/projected/b333236c-930f-4ac0-b950-b02e8e7374a9-kube-api-access-q5lzx\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.728994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-script-lib\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-slash\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-modprobe-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-slash\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-bin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.729221 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-dbus\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/291e2747-1e3c-46df-bd4e-91631ff384f1-kube-api-access-hznjh\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-cnibin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-multus\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-log-socket\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-systemd-units\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-hosts-file\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-os-release\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-kubelet\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-k8s-cni-cncf-io\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-netd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-netns\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-config\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-multus\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.729730 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-run-ovn\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovn-node-metrics-cert\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-env-overrides\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-run\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-daemon-config\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-ovnkube-config\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-modprobe-d\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b333236c-930f-4ac0-b950-b02e8e7374a9-iptables-alerter-script\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.729437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-sys\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.730456 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-kubelet\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-systemd\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-cni-dir\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-cnibin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8b3873af-4061-4796-ab9a-5c9d44bc5675-multus-daemon-config\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-log-socket\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-var-lib-cni-bin\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-os-release\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76047ac2-1c2f-4d1b-be99-9173a606cbef-os-release\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.730928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.730777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-etc-tuned\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.731390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.731390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-env-overrides\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.731390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/785026c7-2c37-44f5-82ae-cc32f614b0d0-dbus\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:11.731390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-cni-netd\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-host-run-multus-certs\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8b3873af-4061-4796-ab9a-5c9d44bc5675-hostroot\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.731497    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-hosts-file\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.731641 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.731590    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:12.231561929 +0000 UTC m=+2.066307084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.731957 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-run\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.732014 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.731952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76047ac2-1c2f-4d1b-be99-9173a606cbef-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.733045 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.733016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-tmp\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.735113 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.735087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fsc\" (UniqueName: \"kubernetes.io/projected/8b3873af-4061-4796-ab9a-5c9d44bc5675-kube-api-access-b9fsc\") pod \"multus-7qmmz\" (UID: \"8b3873af-4061-4796-ab9a-5c9d44bc5675\") " pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.735295 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.735271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldf4\" (UniqueName: \"kubernetes.io/projected/80907e86-b3f6-44a8-ae24-dbbb6aae9f57-kube-api-access-6ldf4\") pod \"tuned-sx2xr\" (UID: \"80907e86-b3f6-44a8-ae24-dbbb6aae9f57\") " pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.738969 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.738931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdpl\" (UniqueName: \"kubernetes.io/projected/3da7d8a9-e9d0-48ab-bccd-bf13478d18f6-kube-api-access-bcdpl\") pod \"ovnkube-node-9jw6s\" (UID: \"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.741003 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.740979    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:11.741126 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.741027    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:11.741126 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.741045    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:11.741245 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:11.741196    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:12.241173593 +0000 UTC m=+2.075918765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:11.743068 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.743043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5m8m\" (UniqueName: \"kubernetes.io/projected/3c2cfa72-ee71-41b3-9313-618a23234d37-kube-api-access-d5m8m\") pod \"node-ca-8ncht\" (UID: \"3c2cfa72-ee71-41b3-9313-618a23234d37\") " pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.743411 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.743389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvj7\" (UniqueName: \"kubernetes.io/projected/76047ac2-1c2f-4d1b-be99-9173a606cbef-kube-api-access-7hvj7\") pod \"multus-additional-cni-plugins-bvm2g\" (UID: \"76047ac2-1c2f-4d1b-be99-9173a606cbef\") " pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.743773 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.743751    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lzx\" (UniqueName: \"kubernetes.io/projected/b333236c-930f-4ac0-b950-b02e8e7374a9-kube-api-access-q5lzx\") pod \"iptables-alerter-rpf64\" (UID: \"b333236c-930f-4ac0-b950-b02e8e7374a9\") " pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.744018 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.744001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdlhs\" (UniqueName: \"kubernetes.io/projected/10a4c5eb-9180-464c-85fe-4d7a009b5ab3-kube-api-access-fdlhs\") pod \"node-resolver-crz4p\" (UID: \"10a4c5eb-9180-464c-85fe-4d7a009b5ab3\") " pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.744163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.744146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznjh\" (UniqueName: \"kubernetes.io/projected/291e2747-1e3c-46df-bd4e-91631ff384f1-kube-api-access-hznjh\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:11.834261 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.834159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:11.842850 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.842823    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bbcca4_dfd3_4609_b22f_88d9d002ae80.slice/crio-3254c81d40c3ccb19394513b7f72a72fc3afd9d267b0f2ec8ae8ba1ed4fcf577 WatchSource:0}: Error finding container 3254c81d40c3ccb19394513b7f72a72fc3afd9d267b0f2ec8ae8ba1ed4fcf577: Status 404 returned error can't find the container with id 3254c81d40c3ccb19394513b7f72a72fc3afd9d267b0f2ec8ae8ba1ed4fcf577
Jul 28 12:35:11.865670 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.865640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d"
Jul 28 12:35:11.873981 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.873931    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16365c9c_4da8_49fb_80d1_5974d4a7d3b3.slice/crio-765a40391784b06a3ff4cc63d52a575c8b67015954010e468003eb37dfb5ee01 WatchSource:0}: Error finding container 765a40391784b06a3ff4cc63d52a575c8b67015954010e468003eb37dfb5ee01: Status 404 returned error can't find the container with id 765a40391784b06a3ff4cc63d52a575c8b67015954010e468003eb37dfb5ee01
Jul 28 12:35:11.881338 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.881316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sx2xr"
Jul 28 12:35:11.888634 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.888608    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80907e86_b3f6_44a8_ae24_dbbb6aae9f57.slice/crio-15f5144b8f262f58911be226925cb830f9c657960efaaee94f79f2c97d8c742a WatchSource:0}: Error finding container 15f5144b8f262f58911be226925cb830f9c657960efaaee94f79f2c97d8c742a: Status 404 returned error can't find the container with id 15f5144b8f262f58911be226925cb830f9c657960efaaee94f79f2c97d8c742a
Jul 28 12:35:11.889333 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.889317    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:11.899922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.899899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crz4p"
Jul 28 12:35:11.922220 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.922190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8ncht"
Jul 28 12:35:11.929552 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.929523    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2cfa72_ee71_41b3_9313_618a23234d37.slice/crio-bebcf7ac28c5db522a84607a6b8cb0c8fc36e790df1988c6e8273be4b31ae1b6 WatchSource:0}: Error finding container bebcf7ac28c5db522a84607a6b8cb0c8fc36e790df1988c6e8273be4b31ae1b6: Status 404 returned error can't find the container with id bebcf7ac28c5db522a84607a6b8cb0c8fc36e790df1988c6e8273be4b31ae1b6
Jul 28 12:35:11.943341 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.943318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qmmz"
Jul 28 12:35:11.951423 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.951395    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3873af_4061_4796_ab9a_5c9d44bc5675.slice/crio-0e678c337a4e722c6aa0d33a752d334b30eec6374be85ab5916d30274ae7d017 WatchSource:0}: Error finding container 0e678c337a4e722c6aa0d33a752d334b30eec6374be85ab5916d30274ae7d017: Status 404 returned error can't find the container with id 0e678c337a4e722c6aa0d33a752d334b30eec6374be85ab5916d30274ae7d017
Jul 28 12:35:11.967134 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.967112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvm2g"
Jul 28 12:35:11.975517 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.975490    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76047ac2_1c2f_4d1b_be99_9173a606cbef.slice/crio-38593f6723c9a1ca36a70b07e52479466dbc5af814fa1f0d81ae8204f555249f WatchSource:0}: Error finding container 38593f6723c9a1ca36a70b07e52479466dbc5af814fa1f0d81ae8204f555249f: Status 404 returned error can't find the container with id 38593f6723c9a1ca36a70b07e52479466dbc5af814fa1f0d81ae8204f555249f
Jul 28 12:35:11.987075 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.987048    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rpf64"
Jul 28 12:35:11.993357 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:11.993313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:11.995965 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:11.995920    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb333236c_930f_4ac0_b950_b02e8e7374a9.slice/crio-079762e9e46983227614e4eaf43a9945eda69f9a04864ba1276b03f91a86b0d0 WatchSource:0}: Error finding container 079762e9e46983227614e4eaf43a9945eda69f9a04864ba1276b03f91a86b0d0: Status 404 returned error can't find the container with id 079762e9e46983227614e4eaf43a9945eda69f9a04864ba1276b03f91a86b0d0
Jul 28 12:35:12.001030 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:12.001007    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da7d8a9_e9d0_48ab_bccd_bf13478d18f6.slice/crio-1df144f61b05a995f1126aa032ecc3c8c5d9a96cf91dc20031cfbb18a00f6021 WatchSource:0}: Error finding container 1df144f61b05a995f1126aa032ecc3c8c5d9a96cf91dc20031cfbb18a00f6021: Status 404 returned error can't find the container with id 1df144f61b05a995f1126aa032ecc3c8c5d9a96cf91dc20031cfbb18a00f6021
Jul 28 12:35:12.233314 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.233274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:12.233528 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.233353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:12.233528 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.233481    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:12.233639 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.233545    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.233524179 +0000 UTC m=+3.068269354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:12.234036 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.234012    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:12.234179 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.234076    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.234058911 +0000 UTC m=+3.068804080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:12.335389 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.334067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:12.335389 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.334265    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:12.335389 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.334286    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:12.335389 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.334302    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:12.335389 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.334371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.33435205 +0000 UTC m=+3.169097210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:12.562752 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.562561    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:30:11 +0000 UTC" deadline="2028-03-10 09:10:07.886238335 +0000 UTC"
Jul 28 12:35:12.562752 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.562630    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14180h34m55.3236126s"
Jul 28 12:35:12.656373 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.655835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:12.656373 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:12.655985    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:12.678329 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.678212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rpf64" event={"ID":"b333236c-930f-4ac0-b950-b02e8e7374a9","Type":"ContainerStarted","Data":"079762e9e46983227614e4eaf43a9945eda69f9a04864ba1276b03f91a86b0d0"}
Jul 28 12:35:12.687665 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.687559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerStarted","Data":"38593f6723c9a1ca36a70b07e52479466dbc5af814fa1f0d81ae8204f555249f"}
Jul 28 12:35:12.702225 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.702181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qmmz" event={"ID":"8b3873af-4061-4796-ab9a-5c9d44bc5675","Type":"ContainerStarted","Data":"0e678c337a4e722c6aa0d33a752d334b30eec6374be85ab5916d30274ae7d017"}
Jul 28 12:35:12.739486 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.739440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8ncht" event={"ID":"3c2cfa72-ee71-41b3-9313-618a23234d37","Type":"ContainerStarted","Data":"bebcf7ac28c5db522a84607a6b8cb0c8fc36e790df1988c6e8273be4b31ae1b6"}
Jul 28 12:35:12.757316 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.757276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crz4p" event={"ID":"10a4c5eb-9180-464c-85fe-4d7a009b5ab3","Type":"ContainerStarted","Data":"c6cb6301b550323e8838da6a0b8a53ba3efc1528241c0a8d4f2ffeb31dcfe4c0"}
Jul 28 12:35:12.799328 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.799271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sx2xr" event={"ID":"80907e86-b3f6-44a8-ae24-dbbb6aae9f57","Type":"ContainerStarted","Data":"15f5144b8f262f58911be226925cb830f9c657960efaaee94f79f2c97d8c742a"}
Jul 28 12:35:12.822415 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.822234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d" event={"ID":"16365c9c-4da8-49fb-80d1-5974d4a7d3b3","Type":"ContainerStarted","Data":"765a40391784b06a3ff4cc63d52a575c8b67015954010e468003eb37dfb5ee01"}
Jul 28 12:35:12.846782 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.846678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs7wx" event={"ID":"f5bbcca4-dfd3-4609-b22f-88d9d002ae80","Type":"ContainerStarted","Data":"3254c81d40c3ccb19394513b7f72a72fc3afd9d267b0f2ec8ae8ba1ed4fcf577"}
Jul 28 12:35:12.860848 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.860588    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:12.862540 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.862406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"1df144f61b05a995f1126aa032ecc3c8c5d9a96cf91dc20031cfbb18a00f6021"}
Jul 28 12:35:12.880870 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.880829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" event={"ID":"61ce3bf83fccda127909e8840dbc1840","Type":"ContainerStarted","Data":"5c746ac3e59102e25abcccb1308824e3a96e5308e352f5010d2ae0418e1d085c"}
Jul 28 12:35:12.914852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:12.914719    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"d10ec0ce97f3475d94c29e5baee4c519","Type":"ContainerStarted","Data":"1e9aa54c74466b277ab9dc9761dec7407bad6085b436efce5a60aa582fcd3206"}
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.242387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.242461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.242713    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.242783    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:15.242763038 +0000 UTC m=+5.077508211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.243220    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:13.243306 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.243269    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:15.243252896 +0000 UTC m=+5.077998055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:13.343125 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.343084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:13.343342 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.343323    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:13.343407 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.343346    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:13.343407 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.343359    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:13.343515 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.343431    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:15.343408101 +0000 UTC m=+5.178153276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:13.563492 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.563338    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:30:11 +0000 UTC" deadline="2028-04-29 13:56:40.564602596 +0000 UTC"
Jul 28 12:35:13.563492 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.563407    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15385h21m27.001200216s"
Jul 28 12:35:13.656331 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.656280    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:13.656331 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:13.656303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:13.656592 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.656431    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:13.656653 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:13.656605    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:14.461964 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:14.461932    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:14.658993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:14.658961    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:14.659470 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:14.659105    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:15.260251 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:15.260199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:15.260452 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:15.260279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:15.260523 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.260456    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:15.260577 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.260522    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:19.260501839 +0000 UTC m=+9.095246994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:15.261034 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.261012    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:15.261145 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.261074    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:19.26105836 +0000 UTC m=+9.095803518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:15.361359 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:15.361322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:15.361553 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.361542    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:15.361623 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.361565    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:15.361623 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.361579    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:15.361729 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.361642    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:19.361620266 +0000 UTC m=+9.196365423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:15.656505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:15.656416    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:15.656505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:15.656453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:15.656751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.656576    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:15.656751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:15.656684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:16.656904 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:16.656736    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:16.657368 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:16.656904    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:17.656684 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:17.656654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:17.656684 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:17.656681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:17.656971 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:17.656781    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:17.656971 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:17.656886    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:18.658531 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:18.657793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:18.658531 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:18.657956    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:19.292690 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:19.292646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:19.292892 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:19.292726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:19.292975 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.292890    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:19.292975 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.292955    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.292935146 +0000 UTC m=+17.127680319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:19.292975 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.292959    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:19.293122 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.292997    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.292985936 +0000 UTC m=+17.127731092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:19.393560 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:19.393425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:19.393751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.393583    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:19.393751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.393604    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:19.393751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.393617    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:19.393751 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.393678    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:27.393660676 +0000 UTC m=+17.228405844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:19.656250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:19.656146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:19.656405 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.656272    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:19.656641 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:19.656143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:19.656641 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:19.656601    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:20.658197 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:20.657741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:20.658197 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:20.657889    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:21.656390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:21.656349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:21.656583 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:21.656406    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:21.656583 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:21.656494    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:21.656769 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:21.656741    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:22.656547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:22.656299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:22.656986 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:22.656678    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:23.656858 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:23.656771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:23.657346 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:23.656771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:23.657346 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:23.656927    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:23.657346 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:23.657000    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:24.656446 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:24.656409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:24.656615 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:24.656542    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:25.656703 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:25.656672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:25.657198 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:25.656797    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:25.657198 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:25.656672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:25.657324 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:25.657289    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:26.656351 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:26.656316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:26.656548 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:26.656464    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:27.354963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:27.354924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:27.355435 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:27.354992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:27.355435 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.355094    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:27.355435 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.355114    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:27.355435 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.355168    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:43.355146636 +0000 UTC m=+33.189891791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:27.355435 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.355193    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:43.35518172 +0000 UTC m=+33.189926877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:27.456032 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:27.456000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:27.456237 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.456188    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:27.456237 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.456213    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:27.456237 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.456225    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:27.456395 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.456291    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:35:43.456271952 +0000 UTC m=+33.291017105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:27.655980 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:27.655945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:27.656154 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:27.655945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:27.656154 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.656080    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:27.656253 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:27.656176    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:28.656946 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:28.656909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:28.657514 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:28.657049    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:29.656372 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:29.656329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:29.656583 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:29.656476    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:29.656583 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:29.656541    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:29.656703 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:29.656679    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:30.657216 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:30.657179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:30.657707 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:30.657299    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:31.656161 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:31.656123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:31.656335 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:31.656128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:31.656335 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:31.656227    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:31.656335 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:31.656319    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:32.656483 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.656305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:32.657122 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:32.656571    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:32.977552 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.977284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" event={"ID":"61ce3bf83fccda127909e8840dbc1840","Type":"ContainerStarted","Data":"371e62a1050b8acdd0a7850b47f05f291e77b2eb4380d77f727de52accffd34b"}
Jul 28 12:35:32.978781 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.978756    2560 generic.go:356] "Generic (PLEG): container finished" podID="d10ec0ce97f3475d94c29e5baee4c519" containerID="376882f6404a7538b68b9afac2b5982302e8e4205b2f4e18e5428ef0afbf287c" exitCode=0
Jul 28 12:35:32.978958 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.978828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"d10ec0ce97f3475d94c29e5baee4c519","Type":"ContainerDied","Data":"376882f6404a7538b68b9afac2b5982302e8e4205b2f4e18e5428ef0afbf287c"}
Jul 28 12:35:32.980289 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.980269    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="61a1ea1a6945a6c5ff41bf3fd4adb23113ca5d43dca891f90558f4372c5e97ca" exitCode=0
Jul 28 12:35:32.980383 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.980332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"61a1ea1a6945a6c5ff41bf3fd4adb23113ca5d43dca891f90558f4372c5e97ca"}
Jul 28 12:35:32.981849 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.981828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qmmz" event={"ID":"8b3873af-4061-4796-ab9a-5c9d44bc5675","Type":"ContainerStarted","Data":"200e52b0adf1ef843e4bd646a61c22f69cea80a63c6e48b6bf135760f6cbe334"}
Jul 28 12:35:32.983222 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.983196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8ncht" event={"ID":"3c2cfa72-ee71-41b3-9313-618a23234d37","Type":"ContainerStarted","Data":"927b1156796578a374d4a206e5cab169d49f394044a0e808cbb1c17e9e048f77"}
Jul 28 12:35:32.984461 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.984442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crz4p" event={"ID":"10a4c5eb-9180-464c-85fe-4d7a009b5ab3","Type":"ContainerStarted","Data":"56f09a331927d6913de03aa57e4da44728de4bb03d17cdcb697dba446cce715a"}
Jul 28 12:35:32.985822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.985799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sx2xr" event={"ID":"80907e86-b3f6-44a8-ae24-dbbb6aae9f57","Type":"ContainerStarted","Data":"ade89b10997dd3ef5c9bf1579758f671c5f4a1d2d9578c360ed3854f11b5fd3c"}
Jul 28 12:35:32.989054 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.989033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d" event={"ID":"16365c9c-4da8-49fb-80d1-5974d4a7d3b3","Type":"ContainerStarted","Data":"0ae9e708fe2444152c1b818959cb89879414d7bbbaa10e24aaf54f595ee69f56"}
Jul 28 12:35:32.990527 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.990505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs7wx" event={"ID":"f5bbcca4-dfd3-4609-b22f-88d9d002ae80","Type":"ContainerStarted","Data":"15f413be8fb9b66dd3daa3e2482b75e6329d821f573bcbdcbd6d76a132a5ea7f"}
Jul 28 12:35:32.993306 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"79bae919d2b7113ca6f0cedd5065c637d4b1e5e921300b0162aef30c057357d9"}
Jul 28 12:35:32.993396 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"aed39136dbe1fdf185b801503b96c5469e16a792d9c600f52e80018b2a8acc40"}
Jul 28 12:35:32.993396 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"196f01e65802da9a5084275034580c36507c012b7ac97ed40338d6eb0db523f2"}
Jul 28 12:35:32.993464 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"034ee9381a36800f4af63623ef438d8b2a6166f4107002a2c72bc2d1f092df52"}
Jul 28 12:35:32.993464 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"e612a328bc14245fec89b7064e48a17702c4670e8fee9c8a3fdc803256153fc9"}
Jul 28 12:35:32.993464 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:32.993452    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"90ecb50b05a834f30daf19ebb18f5445c7fe1d95f0dd9a33e4f4e4143a1bc2bc"}
Jul 28 12:35:33.034264 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.034152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" podStartSLOduration=22.034135026 podStartE2EDuration="22.034135026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.012813325 +0000 UTC m=+22.847558499" watchObservedRunningTime="2026-07-28 12:35:33.034135026 +0000 UTC m=+22.868880197"
Jul 28 12:35:33.053002 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.052954    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vs7wx" podStartSLOduration=1.8945184259999999 podStartE2EDuration="22.052936321s" totalImagesPullingTime="20.158417895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.033520913 +0000 UTC m=+22.868266088" watchObservedRunningTime="2026-07-28 12:35:33.052936321 +0000 UTC m=+22.887681485"
Jul 28 12:35:33.053182 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.053147    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-crz4p" podStartSLOduration=1.896201955 podStartE2EDuration="22.053139819s" totalImagesPullingTime="20.156937864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.052551371 +0000 UTC m=+22.887296544" watchObservedRunningTime="2026-07-28 12:35:33.053139819 +0000 UTC m=+22.887884993"
Jul 28 12:35:33.076200 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.076127    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7qmmz" podStartSLOduration=1.8194382 podStartE2EDuration="22.07611422s" totalImagesPullingTime="20.25667602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.075925506 +0000 UTC m=+22.910670767" watchObservedRunningTime="2026-07-28 12:35:33.07611422 +0000 UTC m=+22.910859394"
Jul 28 12:35:33.143320 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.143260    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sx2xr" podStartSLOduration=2.030859873 podStartE2EDuration="22.143244234s" totalImagesPullingTime="20.112384361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.143031287 +0000 UTC m=+22.977776457" watchObservedRunningTime="2026-07-28 12:35:33.143244234 +0000 UTC m=+22.977989409"
Jul 28 12:35:33.164006 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.163958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8ncht" podStartSLOduration=2.092344872 podStartE2EDuration="22.163940732s" totalImagesPullingTime="20.07159586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:33.163929203 +0000 UTC m=+22.998674400" watchObservedRunningTime="2026-07-28 12:35:33.163940732 +0000 UTC m=+22.998685906"
Jul 28 12:35:33.655908 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.655846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:33.656172 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.655847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:33.656172 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:33.655994    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:33.656172 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:33.656069    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:33.997099 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.997053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rpf64" event={"ID":"b333236c-930f-4ac0-b950-b02e8e7374a9","Type":"ContainerStarted","Data":"b86aed89bf0372e8fa42c455be999aa829d6c360beb20292a8a9b9ff862a9fbb"}
Jul 28 12:35:33.999419 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:33.999347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"d10ec0ce97f3475d94c29e5baee4c519","Type":"ContainerStarted","Data":"cd17b2dd378136523ab36a81997a681dd2ccf3daeb7f036a59d30701abab985e"}
Jul 28 12:35:34.016249 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.016182    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rpf64" podStartSLOduration=3.01117578 podStartE2EDuration="23.016159571s" totalImagesPullingTime="20.004983791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:34.015904139 +0000 UTC m=+23.850649333" watchObservedRunningTime="2026-07-28 12:35:34.016159571 +0000 UTC m=+23.850904745"
Jul 28 12:35:34.110140 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.110111    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:35:34.591452 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.591345    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:35:34.110131318Z","UUID":"a57c3ce4-711c-46e1-9178-036aa7e6c7af","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:35:34.593240 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.593214    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:35:34.593396 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.593260    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:35:34.656060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:34.656021    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:34.656267 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:34.656154    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:35.004054 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.004017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d" event={"ID":"16365c9c-4da8-49fb-80d1-5974d4a7d3b3","Type":"ContainerStarted","Data":"ced905371112d95f73bd6fe729357f9ef7c84a03cf44d8537221f9d564071e64"}
Jul 28 12:35:35.656251 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.656220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:35.656421 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:35.656339    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:35.656421 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.656389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:35.656548 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:35.656497    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:35.776465 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.776375    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:35.777039 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.777018    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:35.794187 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:35.794125    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" podStartSLOduration=24.794100028 podStartE2EDuration="24.794100028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:34.048914049 +0000 UTC m=+23.883659243" watchObservedRunningTime="2026-07-28 12:35:35.794100028 +0000 UTC m=+25.628845201"
Jul 28 12:35:36.009380 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:36.009337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d" event={"ID":"16365c9c-4da8-49fb-80d1-5974d4a7d3b3","Type":"ContainerStarted","Data":"240e926a0c77b9bf4b3cbcd957bf32ccee075dd1f26507b732defa7b6a8d189e"}
Jul 28 12:35:36.013386 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:36.013346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"fb39bb2bf42cd90d4073d4f6a09b995eb982ece8711aa37ed5aab65b9b6b35d7"}
Jul 28 12:35:36.030663 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:36.030539    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnz6d" podStartSLOduration=2.000973062 podStartE2EDuration="25.030519668s" totalImagesPullingTime="23.029546606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:36.030378179 +0000 UTC m=+25.865123351" watchObservedRunningTime="2026-07-28 12:35:36.030519668 +0000 UTC m=+25.865264843"
Jul 28 12:35:36.656291 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:36.656185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:36.656488 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:36.656338    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:37.656072 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:37.655891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:37.656723 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:37.655945    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:37.656723 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:37.656172    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:37.656723 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:37.656229    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:38.021355 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.021323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" event={"ID":"3da7d8a9-e9d0-48ab-bccd-bf13478d18f6","Type":"ContainerStarted","Data":"eb2a5f362ccd1fa39bb94eda39b98bbc9341f4b1aff156a1ca62d12fcd66f1c4"}
Jul 28 12:35:38.021570 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.021548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:38.023030 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.023003    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="f1acf9e085aca7f84d72f2f8a42a0a0b4d65b185bbde2537b5e0a09106a63781" exitCode=0
Jul 28 12:35:38.023143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.023047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"f1acf9e085aca7f84d72f2f8a42a0a0b4d65b185bbde2537b5e0a09106a63781"}
Jul 28 12:35:38.037527 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.037488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:38.052477 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.052428    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s" podStartSLOduration=6.96989394 podStartE2EDuration="27.052412374s" totalImagesPullingTime="20.082518434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:38.052003745 +0000 UTC m=+27.886748942" watchObservedRunningTime="2026-07-28 12:35:38.052412374 +0000 UTC m=+27.887157548"
Jul 28 12:35:38.462017 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.461968    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:38.660106 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:38.660077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:38.660842 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:38.660189    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:39.033587 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.033419    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:39.052085 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.052055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:35:39.656889 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.656831    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:39.657066 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.656831    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:39.657066 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:39.656969    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:39.657066 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:39.657015    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:39.716698 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.716662    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86zqw"]
Jul 28 12:35:39.717117 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.716805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:39.717117 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:39.716926    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:39.719943 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.719906    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ksczq"]
Jul 28 12:35:39.720584 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:39.720557    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-llr2w"]
Jul 28 12:35:40.039993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:40.039943    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="db1df2b6d256e112a4686e98570f3985b9f462248787d16747c74425821d4a08" exitCode=0
Jul 28 12:35:40.039993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:40.039984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"db1df2b6d256e112a4686e98570f3985b9f462248787d16747c74425821d4a08"}
Jul 28 12:35:40.040193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:40.040077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:40.040193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:40.040077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:40.040257 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:40.040182    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:40.040513 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:40.040489    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:41.027133 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.027089    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:41.027642 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.027224    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 12:35:41.027761 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.027745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vs7wx"
Jul 28 12:35:41.656252 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.656217    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:41.656416 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.656256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:41.656416 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:41.656270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:41.656416 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:41.656359    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:41.656586 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:41.656481    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:41.656586 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:41.656554    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:42.046272 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:42.046183    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="fed9473de2b9b1d95af19ca4c3fcfd23970e6818afbb43a5ffee9360faf413cf" exitCode=0
Jul 28 12:35:42.046272 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:42.046231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"fed9473de2b9b1d95af19ca4c3fcfd23970e6818afbb43a5ffee9360faf413cf"}
Jul 28 12:35:43.375015 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.374683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:43.375450 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.374837    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:43.375450 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.375088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:43.375450 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.375162    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs podName:291e2747-1e3c-46df-bd4e-91631ff384f1 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:15.375137484 +0000 UTC m=+65.209882636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs") pod "network-metrics-daemon-86zqw" (UID: "291e2747-1e3c-46df-bd4e-91631ff384f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:43.375450 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.375229    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:43.375450 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.375285    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret podName:785026c7-2c37-44f5-82ae-cc32f614b0d0 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:15.375273262 +0000 UTC m=+65.210018413 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret") pod "global-pull-secret-syncer-llr2w" (UID: "785026c7-2c37-44f5-82ae-cc32f614b0d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:43.476489 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.476451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:43.476677 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.476637    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:43.476677 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.476667    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:43.476769 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.476683    2560 projected.go:196] Error preparing data for projected volume kube-api-access-mc6gz for pod openshift-network-diagnostics/network-check-target-ksczq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:43.476769 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.476749    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz podName:28edcfdd-cbed-4aad-b0fb-d2d1677c1adc nodeName:}" failed. No retries permitted until 2026-07-28 12:36:15.476728491 +0000 UTC m=+65.311473660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mc6gz" (UniqueName: "kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz") pod "network-check-target-ksczq" (UID: "28edcfdd-cbed-4aad-b0fb-d2d1677c1adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:43.656305 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.656221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:43.656465 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.656229    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:43.656465 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:43.656352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:43.656465 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.656350    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksczq" podUID="28edcfdd-cbed-4aad-b0fb-d2d1677c1adc"
Jul 28 12:35:43.656465 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.656431    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-llr2w" podUID="785026c7-2c37-44f5-82ae-cc32f614b0d0"
Jul 28 12:35:43.656655 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:43.656525    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86zqw" podUID="291e2747-1e3c-46df-bd4e-91631ff384f1"
Jul 28 12:35:44.977345 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:44.977311    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeReady"
Jul 28 12:35:44.977765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:44.977481    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:35:45.054934 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.054894    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nkh8w"]
Jul 28 12:35:45.057003 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.056976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.058734 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.058712    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jznjs"]
Jul 28 12:35:45.060568 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.060547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.063550 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063507    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:35:45.063550 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063530    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:35:45.063757 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063657    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:35:45.063816 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063789    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:35:45.063910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063810    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k2njb\""
Jul 28 12:35:45.063910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.063894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:35:45.064558 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.064538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l7wd\""
Jul 28 12:35:45.075612 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.075311    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nkh8w"]
Jul 28 12:35:45.076172 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.076150    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jznjs"]
Jul 28 12:35:45.188937 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.188891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f933002c-75ff-463e-90c2-e42816673f7f-tmp-dir\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.189095 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.188959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922664ba-bfbd-4fa5-8d4f-6104e46d0567-cert\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.189095 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.189065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r95\" (UniqueName: \"kubernetes.io/projected/922664ba-bfbd-4fa5-8d4f-6104e46d0567-kube-api-access-l8r95\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.189215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.189094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f933002c-75ff-463e-90c2-e42816673f7f-config-volume\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.189215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.189124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.189215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.189170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xjt\" (UniqueName: \"kubernetes.io/projected/f933002c-75ff-463e-90c2-e42816673f7f-kube-api-access-58xjt\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290259 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f933002c-75ff-463e-90c2-e42816673f7f-tmp-dir\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290259 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922664ba-bfbd-4fa5-8d4f-6104e46d0567-cert\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.290462 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r95\" (UniqueName: \"kubernetes.io/projected/922664ba-bfbd-4fa5-8d4f-6104e46d0567-kube-api-access-l8r95\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.290462 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f933002c-75ff-463e-90c2-e42816673f7f-config-volume\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290462 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290462 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xjt\" (UniqueName: \"kubernetes.io/projected/f933002c-75ff-463e-90c2-e42816673f7f-kube-api-access-58xjt\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290648 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.290581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f933002c-75ff-463e-90c2-e42816673f7f-tmp-dir\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.290821 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:45.290803    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:45.290896 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:45.290888    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:45.790854063 +0000 UTC m=+35.625599215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:45.291091 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.291069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f933002c-75ff-463e-90c2-e42816673f7f-config-volume\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.295473 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.295438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922664ba-bfbd-4fa5-8d4f-6104e46d0567-cert\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.300714 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.300680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xjt\" (UniqueName: \"kubernetes.io/projected/f933002c-75ff-463e-90c2-e42816673f7f-kube-api-access-58xjt\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.300899 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.300770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r95\" (UniqueName: \"kubernetes.io/projected/922664ba-bfbd-4fa5-8d4f-6104e46d0567-kube-api-access-l8r95\") pod \"ingress-canary-jznjs\" (UID: \"922664ba-bfbd-4fa5-8d4f-6104e46d0567\") " pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.378811 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.378773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jznjs"
Jul 28 12:35:45.656885 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.656791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:35:45.656885 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.656815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:35:45.657107 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.656920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:35:45.659732 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.659703    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:35:45.659887 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.659748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z4tgw\""
Jul 28 12:35:45.659999 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.659987    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:35:45.660913 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.660840    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:35:45.661401 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.661370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:35:45.661500 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.661397    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kxg2r\""
Jul 28 12:35:45.794242 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:45.794202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:45.794433 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:45.794345    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:45.794494 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:45.794473    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:46.794452535 +0000 UTC m=+36.629197690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:46.743437 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:46.743236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jznjs"]
Jul 28 12:35:46.749373 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:46.749323    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922664ba_bfbd_4fa5_8d4f_6104e46d0567.slice/crio-9c261aadc4858d9915f7de34ea2c1be2d3f6a68dc954723e85797f940d18af39 WatchSource:0}: Error finding container 9c261aadc4858d9915f7de34ea2c1be2d3f6a68dc954723e85797f940d18af39: Status 404 returned error can't find the container with id 9c261aadc4858d9915f7de34ea2c1be2d3f6a68dc954723e85797f940d18af39
Jul 28 12:35:46.804910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:46.804854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:46.805052 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:46.804977    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:46.805181 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:46.805054    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:48.805038076 +0000 UTC m=+38.639783229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:47.059097 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.059055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jznjs" event={"ID":"922664ba-bfbd-4fa5-8d4f-6104e46d0567","Type":"ContainerStarted","Data":"9c261aadc4858d9915f7de34ea2c1be2d3f6a68dc954723e85797f940d18af39"}
Jul 28 12:35:47.061716 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.061685    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="02a53f7f49fcc9f92725cf162365401299015452856dbfb8cef9d309a59848d0" exitCode=0
Jul 28 12:35:47.061893 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.061734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"02a53f7f49fcc9f92725cf162365401299015452856dbfb8cef9d309a59848d0"}
Jul 28 12:35:47.248362 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.248331    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"]
Jul 28 12:35:47.250106 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.250090    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.256792 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.256773    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 12:35:47.257187 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.257163    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 12:35:47.257273 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.257227    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8bnbx\""
Jul 28 12:35:47.258065 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.258049    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.261192 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.261176    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.270777 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.270742    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"]
Jul 28 12:35:47.271268 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.271249    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"]
Jul 28 12:35:47.273287 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.273265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"
Jul 28 12:35:47.281606 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.281583    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.282195 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.282168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kd6xx\""
Jul 28 12:35:47.282559 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.282547    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.293673 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.293640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"]
Jul 28 12:35:47.354377 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.354298    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-66s5m"]
Jul 28 12:35:47.356122 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.356106    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.360812 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.360788    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.362276 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.361997    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8cv4b\""
Jul 28 12:35:47.363206 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.363181    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 12:35:47.363765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.363747    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.364068 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.364050    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 12:35:47.368301 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.368272    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 12:35:47.381699 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.381662    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-66s5m"]
Jul 28 12:35:47.408975 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.408932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.409167 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.408980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhtt\" (UniqueName: \"kubernetes.io/projected/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-kube-api-access-cbhtt\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.409167 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.409048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpf95\" (UniqueName: \"kubernetes.io/projected/22eb831c-1907-4971-b149-5a1e652035cd-kube-api-access-rpf95\") pod \"volume-data-source-validator-549c6d54cf-bcjj5\" (UID: \"22eb831c-1907-4971-b149-5a1e652035cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"
Jul 28 12:35:47.409167 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.409069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.451540 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.451505    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f7786d8cf-rm5kx"]
Jul 28 12:35:47.453451 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.453435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.457989 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.457964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 12:35:47.458133 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.457963    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:35:47.458301 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.458287    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 12:35:47.460342 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.460321    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-86rvp\""
Jul 28 12:35:47.460476 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.460458    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 12:35:47.465659 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.465640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 12:35:47.466324 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.466308    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:35:47.481679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.481645    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f7786d8cf-rm5kx"]
Jul 28 12:35:47.509767 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.509724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f811ac55-6afa-4b88-bf84-58957e05b5c2-serving-cert\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.509767 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.509764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.510035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.509795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhtt\" (UniqueName: \"kubernetes.io/projected/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-kube-api-access-cbhtt\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.510035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.509857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-config\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.510035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.509934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54s8d\" (UniqueName: \"kubernetes.io/projected/f811ac55-6afa-4b88-bf84-58957e05b5c2-kube-api-access-54s8d\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.510035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.510027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpf95\" (UniqueName: \"kubernetes.io/projected/22eb831c-1907-4971-b149-5a1e652035cd-kube-api-access-rpf95\") pod \"volume-data-source-validator-549c6d54cf-bcjj5\" (UID: \"22eb831c-1907-4971-b149-5a1e652035cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"
Jul 28 12:35:47.510189 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.510058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.510189 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.510118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-trusted-ca\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.510351 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.510335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.513158 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.513139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.527095 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.527058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhtt\" (UniqueName: \"kubernetes.io/projected/dcd4d309-870a-4c7c-b841-bcd5bd5235e5-kube-api-access-cbhtt\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm\" (UID: \"dcd4d309-870a-4c7c-b841-bcd5bd5235e5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.527245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.527231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpf95\" (UniqueName: \"kubernetes.io/projected/22eb831c-1907-4971-b149-5a1e652035cd-kube-api-access-rpf95\") pod \"volume-data-source-validator-549c6d54cf-bcjj5\" (UID: \"22eb831c-1907-4971-b149-5a1e652035cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"
Jul 28 12:35:47.559212 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.559170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"
Jul 28 12:35:47.582349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.582310    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"
Jul 28 12:35:47.611132 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.611132 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.611339 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54s8d\" (UniqueName: \"kubernetes.io/projected/f811ac55-6afa-4b88-bf84-58957e05b5c2-kube-api-access-54s8d\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.611339 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-trusted-ca\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.611339 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-default-certificate\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.611339 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6jc\" (UniqueName: \"kubernetes.io/projected/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-kube-api-access-7s6jc\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.611521 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f811ac55-6afa-4b88-bf84-58957e05b5c2-serving-cert\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.611521 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-stats-auth\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.611521 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.611444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-config\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.612666 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.612472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-config\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.612666 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.612605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f811ac55-6afa-4b88-bf84-58957e05b5c2-trusted-ca\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.615116 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.615048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f811ac55-6afa-4b88-bf84-58957e05b5c2-serving-cert\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.623707 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.623676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54s8d\" (UniqueName: \"kubernetes.io/projected/f811ac55-6afa-4b88-bf84-58957e05b5c2-kube-api-access-54s8d\") pod \"console-operator-584dcb6b54-66s5m\" (UID: \"f811ac55-6afa-4b88-bf84-58957e05b5c2\") " pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.668547 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.668509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:47.700944 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.700897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm"]
Jul 28 12:35:47.706450 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:47.706409    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd4d309_870a_4c7c_b841_bcd5bd5235e5.slice/crio-d3297bd5d6113ad3a00632df5268931ab18f5a392a0043f8675888f66feddece WatchSource:0}: Error finding container d3297bd5d6113ad3a00632df5268931ab18f5a392a0043f8675888f66feddece: Status 404 returned error can't find the container with id d3297bd5d6113ad3a00632df5268931ab18f5a392a0043f8675888f66feddece
Jul 28 12:35:47.712716 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.712682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-default-certificate\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.712821 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.712733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6jc\" (UniqueName: \"kubernetes.io/projected/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-kube-api-access-7s6jc\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.712821 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.712789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-stats-auth\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.713008 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.712986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.713067 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.713021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.713155 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:47.713139    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:48.213118237 +0000 UTC m=+38.047863412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:35:47.713212 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:47.713186    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:35:47.713250 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:47.713223    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:48.213214076 +0000 UTC m=+38.047959229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : secret "router-metrics-certs-default" not found
Jul 28 12:35:47.715523 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.715487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-default-certificate\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.718346 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.718313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-stats-auth\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.725840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.725783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6jc\" (UniqueName: \"kubernetes.io/projected/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-kube-api-access-7s6jc\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:47.728463 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.728432    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5"]
Jul 28 12:35:47.733489 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:47.733453    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eb831c_1907_4971_b149_5a1e652035cd.slice/crio-fb1790b6578b4c770f71850e653091ab7b61739d8dd434da99cb739bc364ee36 WatchSource:0}: Error finding container fb1790b6578b4c770f71850e653091ab7b61739d8dd434da99cb739bc364ee36: Status 404 returned error can't find the container with id fb1790b6578b4c770f71850e653091ab7b61739d8dd434da99cb739bc364ee36
Jul 28 12:35:47.809678 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:47.809649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-66s5m"]
Jul 28 12:35:47.812062 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:47.812027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf811ac55_6afa_4b88_bf84_58957e05b5c2.slice/crio-857ba1e27f4b3eb69fb6f1dbfd2fd83d883b4ba8ab341f3eb9ba36d879e695e0 WatchSource:0}: Error finding container 857ba1e27f4b3eb69fb6f1dbfd2fd83d883b4ba8ab341f3eb9ba36d879e695e0: Status 404 returned error can't find the container with id 857ba1e27f4b3eb69fb6f1dbfd2fd83d883b4ba8ab341f3eb9ba36d879e695e0
Jul 28 12:35:48.068437 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.068403    2560 generic.go:356] "Generic (PLEG): container finished" podID="76047ac2-1c2f-4d1b-be99-9173a606cbef" containerID="c82f509cc2f032be06824910f5277809c629c1a28819833aafdbc31d955565e9" exitCode=0
Jul 28 12:35:48.068621 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.068456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerDied","Data":"c82f509cc2f032be06824910f5277809c629c1a28819833aafdbc31d955565e9"}
Jul 28 12:35:48.069974 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.069941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" event={"ID":"f811ac55-6afa-4b88-bf84-58957e05b5c2","Type":"ContainerStarted","Data":"857ba1e27f4b3eb69fb6f1dbfd2fd83d883b4ba8ab341f3eb9ba36d879e695e0"}
Jul 28 12:35:48.071392 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.071362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm" event={"ID":"dcd4d309-870a-4c7c-b841-bcd5bd5235e5","Type":"ContainerStarted","Data":"d3297bd5d6113ad3a00632df5268931ab18f5a392a0043f8675888f66feddece"}
Jul 28 12:35:48.072687 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.072657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5" event={"ID":"22eb831c-1907-4971-b149-5a1e652035cd","Type":"ContainerStarted","Data":"fb1790b6578b4c770f71850e653091ab7b61739d8dd434da99cb739bc364ee36"}
Jul 28 12:35:48.217784 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.217361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:48.217784 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:48.217584    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:49.217563782 +0000 UTC m=+39.052308933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:35:48.217784 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.217638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:48.218090 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:48.217805    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:35:48.218090 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:48.217886    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:49.21784739 +0000 UTC m=+39.052592549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : secret "router-metrics-certs-default" not found
Jul 28 12:35:48.821224 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:48.821168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:48.821810 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:48.821349    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:48.821810 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:48.821434    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:35:52.821411901 +0000 UTC m=+42.656157058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:49.080303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:49.080219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" event={"ID":"76047ac2-1c2f-4d1b-be99-9173a606cbef","Type":"ContainerStarted","Data":"96806330e3ccf821d9d394ee164add00d8f0849a2616a9c622cc0a6b707fd632"}
Jul 28 12:35:49.110044 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:49.109623    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bvm2g" podStartSLOduration=6.624660632 podStartE2EDuration="38.109600193s" totalImagesPullingTime="31.484939561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:49.107309013 +0000 UTC m=+38.942054214" watchObservedRunningTime="2026-07-28 12:35:49.109600193 +0000 UTC m=+38.944345371"
Jul 28 12:35:49.225330 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:49.225283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:49.225330 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:49.225334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:49.225647 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:49.225579    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:35:49.225647 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:49.225589    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:51.22557179 +0000 UTC m=+41.060316945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:35:49.225647 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:49.225633    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:51.225622074 +0000 UTC m=+41.060367229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : secret "router-metrics-certs-default" not found
Jul 28 12:35:51.245233 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:51.245191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:51.245233 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:51.245230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:51.245708 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:51.245327    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:35:51.245708 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:51.245360    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:55.245342735 +0000 UTC m=+45.080087887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:35:51.245708 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:51.245383    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:55.24537725 +0000 UTC m=+45.080122402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : secret "router-metrics-certs-default" not found
Jul 28 12:35:52.581223 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.581187    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2qkxw"]
Jul 28 12:35:52.601217 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.601184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.604046 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.603952    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:35:52.604181 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.604073    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:35:52.604386 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.604370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:35:52.605201 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.605179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nrjs7\""
Jul 28 12:35:52.605451 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.605435    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:35:52.609712 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.609682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qkxw"]
Jul 28 12:35:52.758074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.758031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.758074 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.758069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.758328 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.758091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36724ce5-e4fa-446d-88de-80d9d42b271d-data-volume\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.758328 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.758135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v856x\" (UniqueName: \"kubernetes.io/projected/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-api-access-v856x\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.758328 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.758317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36724ce5-e4fa-446d-88de-80d9d42b271d-crio-socket\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859114 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36724ce5-e4fa-446d-88de-80d9d42b271d-crio-socket\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859114 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:35:52.859114 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36724ce5-e4fa-446d-88de-80d9d42b271d-crio-socket\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859210    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859212    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859302    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859279    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:53.359257364 +0000 UTC m=+43.194002538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:52.859359 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36724ce5-e4fa-446d-88de-80d9d42b271d-data-volume\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859700 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v856x\" (UniqueName: \"kubernetes.io/projected/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-api-access-v856x\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.859700 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859386    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:53.359370602 +0000 UTC m=+43.194115758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:52.859700 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:52.859403    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.859397628 +0000 UTC m=+50.694142780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:35:52.859700 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.859656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36724ce5-e4fa-446d-88de-80d9d42b271d-data-volume\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:52.871225 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:52.871192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v856x\" (UniqueName: \"kubernetes.io/projected/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-api-access-v856x\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:53.090627 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.090585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jznjs" event={"ID":"922664ba-bfbd-4fa5-8d4f-6104e46d0567","Type":"ContainerStarted","Data":"89f5cd2123a48e7766c61df9ba5f240c4fca0d6a68f5733cefdd92ed09eb9ab7"}
Jul 28 12:35:53.092147 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.092113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5" event={"ID":"22eb831c-1907-4971-b149-5a1e652035cd","Type":"ContainerStarted","Data":"24bb31f78292fd4157f9960257c54116c85806dffa4933391d688a1cbdabfdca"}
Jul 28 12:35:53.093679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.093657    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/0.log"
Jul 28 12:35:53.093791 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.093693    2560 generic.go:356] "Generic (PLEG): container finished" podID="f811ac55-6afa-4b88-bf84-58957e05b5c2" containerID="e8c82d08949acfb3afe799a009d3f861088bad3d64432ab8669aaf224473b59a" exitCode=255
Jul 28 12:35:53.093791 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.093720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" event={"ID":"f811ac55-6afa-4b88-bf84-58957e05b5c2","Type":"ContainerDied","Data":"e8c82d08949acfb3afe799a009d3f861088bad3d64432ab8669aaf224473b59a"}
Jul 28 12:35:53.093967 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.093952    2560 scope.go:117] "RemoveContainer" containerID="e8c82d08949acfb3afe799a009d3f861088bad3d64432ab8669aaf224473b59a"
Jul 28 12:35:53.095262 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.095242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm" event={"ID":"dcd4d309-870a-4c7c-b841-bcd5bd5235e5","Type":"ContainerStarted","Data":"4960699e12e0f9201df84b7f62a758f19ffcdbe285dce37d529c36e446b8ce73"}
Jul 28 12:35:53.112773 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.112745    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7sjgh"]
Jul 28 12:35:53.115653 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.115607    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jznjs" podStartSLOduration=2.568556353 podStartE2EDuration="8.115582389s" totalImagesPullingTime="5.547026036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:53.115003258 +0000 UTC m=+42.949748432" watchObservedRunningTime="2026-07-28 12:35:53.115582389 +0000 UTC m=+42.950327565"
Jul 28 12:35:53.141388 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.141156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7sjgh"]
Jul 28 12:35:53.141388 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.141317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.146161 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.146041    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j4cll\""
Jul 28 12:35:53.146161 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.146135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 12:35:53.146428 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.146131    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 12:35:53.146638 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.146614    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 12:35:53.146748 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.146682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 12:35:53.198545 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.198489    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm" podStartSLOduration=1.609031597 podStartE2EDuration="6.198470631s" totalImagesPullingTime="4.589439034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:53.171752291 +0000 UTC m=+43.006497465" watchObservedRunningTime="2026-07-28 12:35:53.198470631 +0000 UTC m=+43.033215806"
Jul 28 12:35:53.199176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.199136    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bcjj5" podStartSLOduration=1.636067127 podStartE2EDuration="6.19912569s" totalImagesPullingTime="4.563058563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:53.197829912 +0000 UTC m=+43.032575131" watchObservedRunningTime="2026-07-28 12:35:53.19912569 +0000 UTC m=+43.033870842"
Jul 28 12:35:53.262149 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.262097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/41003c71-6657-4f58-b38e-7b4619363af0-signing-cabundle\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.262327 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.262181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/41003c71-6657-4f58-b38e-7b4619363af0-signing-key\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.262391 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.262343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k745n\" (UniqueName: \"kubernetes.io/projected/41003c71-6657-4f58-b38e-7b4619363af0-kube-api-access-k745n\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.363020 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.362911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k745n\" (UniqueName: \"kubernetes.io/projected/41003c71-6657-4f58-b38e-7b4619363af0-kube-api-access-k745n\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.363020 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.362993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:53.363254 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:53.363103    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:53.363254 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:53.363167    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:54.363147237 +0000 UTC m=+44.197892391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:53.363254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.363210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/41003c71-6657-4f58-b38e-7b4619363af0-signing-cabundle\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.363254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.363241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:53.363448 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.363275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/41003c71-6657-4f58-b38e-7b4619363af0-signing-key\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.363448 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:53.363314    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:53.363448 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:53.363369    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:54.363354937 +0000 UTC m=+44.198100093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:53.363953 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.363925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/41003c71-6657-4f58-b38e-7b4619363af0-signing-cabundle\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.365807 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.365781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/41003c71-6657-4f58-b38e-7b4619363af0-signing-key\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.372963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.372932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k745n\" (UniqueName: \"kubernetes.io/projected/41003c71-6657-4f58-b38e-7b4619363af0-kube-api-access-k745n\") pod \"service-ca-658d7b7c78-7sjgh\" (UID: \"41003c71-6657-4f58-b38e-7b4619363af0\") " pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.464163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.464118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh"
Jul 28 12:35:53.614008 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:53.613913    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7sjgh"]
Jul 28 12:35:53.618815 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:35:53.618784    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41003c71_6657_4f58_b38e_7b4619363af0.slice/crio-6d5a2b27ac8cdc2f8f792d23b531c3cb3fbcb8e1ba1ced01622f3f031d7a8c56 WatchSource:0}: Error finding container 6d5a2b27ac8cdc2f8f792d23b531c3cb3fbcb8e1ba1ced01622f3f031d7a8c56: Status 404 returned error can't find the container with id 6d5a2b27ac8cdc2f8f792d23b531c3cb3fbcb8e1ba1ced01622f3f031d7a8c56
Jul 28 12:35:54.099065 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.099024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh" event={"ID":"41003c71-6657-4f58-b38e-7b4619363af0","Type":"ContainerStarted","Data":"6d5a2b27ac8cdc2f8f792d23b531c3cb3fbcb8e1ba1ced01622f3f031d7a8c56"}
Jul 28 12:35:54.100596 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.100576    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:35:54.100975 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.100962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/0.log"
Jul 28 12:35:54.101053 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.100991    2560 generic.go:356] "Generic (PLEG): container finished" podID="f811ac55-6afa-4b88-bf84-58957e05b5c2" containerID="39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e" exitCode=255
Jul 28 12:35:54.101104 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.101089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" event={"ID":"f811ac55-6afa-4b88-bf84-58957e05b5c2","Type":"ContainerDied","Data":"39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e"}
Jul 28 12:35:54.101144 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.101134    2560 scope.go:117] "RemoveContainer" containerID="e8c82d08949acfb3afe799a009d3f861088bad3d64432ab8669aaf224473b59a"
Jul 28 12:35:54.101428 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.101409    2560 scope.go:117] "RemoveContainer" containerID="39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e"
Jul 28 12:35:54.101635 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:54.101612    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-66s5m_openshift-console-operator(f811ac55-6afa-4b88-bf84-58957e05b5c2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" podUID="f811ac55-6afa-4b88-bf84-58957e05b5c2"
Jul 28 12:35:54.373766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.373666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:54.373766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.373738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:54.374023 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:54.373812    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:54.374023 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:54.373857    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:54.374023 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:54.373898    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:56.373880565 +0000 UTC m=+46.208625731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:54.374023 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:54.373932    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:35:56.373916052 +0000 UTC m=+46.208661205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:54.549722 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:54.549692    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crz4p_10a4c5eb-9180-464c-85fe-4d7a009b5ab3/dns-node-resolver/0.log"
Jul 28 12:35:55.106101 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:55.106073    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:35:55.106596 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:55.106521    2560 scope.go:117] "RemoveContainer" containerID="39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e"
Jul 28 12:35:55.106709 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:55.106690    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-66s5m_openshift-console-operator(f811ac55-6afa-4b88-bf84-58957e05b5c2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" podUID="f811ac55-6afa-4b88-bf84-58957e05b5c2"
Jul 28 12:35:55.284020 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:55.283973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:55.284020 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:55.284021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:35:55.284286 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:55.284147    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:35:55.284286 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:55.284157    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:03.284136946 +0000 UTC m=+53.118882100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:35:55.284286 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:55.284192    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:03.284181174 +0000 UTC m=+53.118926339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : secret "router-metrics-certs-default" not found
Jul 28 12:35:55.553476 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:55.553442    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8ncht_3c2cfa72-ee71-41b3-9313-618a23234d37/node-ca/0.log"
Jul 28 12:35:56.351501 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:56.351467    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jznjs_922664ba-bfbd-4fa5-8d4f-6104e46d0567/serve-healthcheck-canary/0.log"
Jul 28 12:35:56.392786 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:56.392745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:56.392970 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:56.392912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:35:56.392970 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:56.392923    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:56.393042 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:56.392988    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.392967797 +0000 UTC m=+50.227712949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:35:56.393042 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:56.393006    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:35:56.393126 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:56.393047    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:00.393035721 +0000 UTC m=+50.227780873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : configmap "kube-rbac-proxy" not found
Jul 28 12:35:57.113217 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:57.113173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh" event={"ID":"41003c71-6657-4f58-b38e-7b4619363af0","Type":"ContainerStarted","Data":"1c32479f9d9422b6fe8dea2c6e15349150211da5d119a17faa80a467921da531"}
Jul 28 12:35:57.135153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:57.135091    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-7sjgh" podStartSLOduration=1.347426048 podStartE2EDuration="4.135073203s" totalImagesPullingTime="2.787647155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:57.133814445 +0000 UTC m=+46.968559641" watchObservedRunningTime="2026-07-28 12:35:57.135073203 +0000 UTC m=+46.969818376"
Jul 28 12:35:57.669628 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:57.669585    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:57.670047 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:57.669671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:35:57.670047 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:35:57.670031    2560 scope.go:117] "RemoveContainer" containerID="39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e"
Jul 28 12:35:57.670204 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:35:57.670183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-66s5m_openshift-console-operator(f811ac55-6afa-4b88-bf84-58957e05b5c2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" podUID="f811ac55-6afa-4b88-bf84-58957e05b5c2"
Jul 28 12:36:00.430746 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:00.430703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:00.431242 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:00.430771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:00.431242 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.430947    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:36:00.431242 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.431010    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.430995941 +0000 UTC m=+58.265741094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : configmap "kube-rbac-proxy" not found
Jul 28 12:36:00.431242 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.430949    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:00.431242 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.431043    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.431036692 +0000 UTC m=+58.265781844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:00.934556 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:00.934509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:00.934750 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.934665    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:36:00.934750 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:00.934734    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls podName:f933002c-75ff-463e-90c2-e42816673f7f nodeName:}" failed. No retries permitted until 2026-07-28 12:36:16.934717512 +0000 UTC m=+66.769462664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls") pod "dns-default-nkh8w" (UID: "f933002c-75ff-463e-90c2-e42816673f7f") : secret "dns-default-metrics-tls" not found
Jul 28 12:36:03.353407 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:03.353354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:03.353407 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:03.353410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:03.353976 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:03.353515    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle podName:9b0a2c52-afb9-4f06-a3a9-5c5416d72895 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:19.353497475 +0000 UTC m=+69.188242627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle") pod "router-default-6f7786d8cf-rm5kx" (UID: "9b0a2c52-afb9-4f06-a3a9-5c5416d72895") : configmap references non-existent config key: service-ca.crt
Jul 28 12:36:03.355859 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:03.355821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-metrics-certs\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:08.495244 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:08.495202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:08.495723 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:08.495361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:08.495723 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:08.495460    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:08.495723 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:08.495519    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls podName:36724ce5-e4fa-446d-88de-80d9d42b271d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:24.495505031 +0000 UTC m=+74.330250183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qkxw" (UID: "36724ce5-e4fa-446d-88de-80d9d42b271d") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:36:08.495723 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:08.495717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36724ce5-e4fa-446d-88de-80d9d42b271d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:11.061769 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:11.061737    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9jw6s"
Jul 28 12:36:12.657002 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:12.656967    2560 scope.go:117] "RemoveContainer" containerID="39c63904e0e2386435ced4e978592c74c47bb8b233371fd3bfc9e7acda7a1d0e"
Jul 28 12:36:13.159341 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.159312    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:36:13.159510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.159364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" event={"ID":"f811ac55-6afa-4b88-bf84-58957e05b5c2","Type":"ContainerStarted","Data":"7b410ab15d663347d7e1f9716b63ea94beb868385a91d4d3c9c5aea7bfa34cf2"}
Jul 28 12:36:13.159668 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.159638    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:36:13.195407 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.195345    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m" podStartSLOduration=21.705915484 podStartE2EDuration="26.19532504s" totalImagesPullingTime="4.489409556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:13.193701486 +0000 UTC m=+63.028446662" watchObservedRunningTime="2026-07-28 12:36:13.19532504 +0000 UTC m=+63.030070280"
Jul 28 12:36:13.231134 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.231091    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"]
Jul 28 12:36:13.233391 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.233372    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.237827 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.237801    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:36:13.238366 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.238349    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:36:13.239043 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.239021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:36:13.239223 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.239197    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 12:36:13.253336 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.253304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"]
Jul 28 12:36:13.275794 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.275761    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bf7d87c9f-99m9w"]
Jul 28 12:36:13.277759 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.277742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.282075 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.282050    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w2rc8\""
Jul 28 12:36:13.282075 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.282051    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:36:13.283135 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.283119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:36:13.284646 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.284580    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:36:13.294595 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.293568    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:36:13.324368 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.324331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf7d87c9f-99m9w"]
Jul 28 12:36:13.331378 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78ba881c-b335-41a8-94c9-99a6291620e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.331566 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnsl\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-kube-api-access-6bnsl\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331566 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ba881c-b335-41a8-94c9-99a6291620e8-tmp\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.331566 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-installation-pull-secrets\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331566 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfmb\" (UniqueName: \"kubernetes.io/projected/78ba881c-b335-41a8-94c9-99a6291620e8-kube-api-access-gxfmb\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.331790 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-image-registry-private-configuration\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331790 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-certificates\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331790 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99dd8cb4-20eb-4851-8301-015e25dac6a6-ca-trust-extracted\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331958 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-tls\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331958 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-bound-sa-token\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.331958 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.331896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-trusted-ca\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432687 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfmb\" (UniqueName: \"kubernetes.io/projected/78ba881c-b335-41a8-94c9-99a6291620e8-kube-api-access-gxfmb\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.432687 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-image-registry-private-configuration\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-certificates\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99dd8cb4-20eb-4851-8301-015e25dac6a6-ca-trust-extracted\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-tls\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-bound-sa-token\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-trusted-ca\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78ba881c-b335-41a8-94c9-99a6291620e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnsl\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-kube-api-access-6bnsl\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.433288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.432984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ba881c-b335-41a8-94c9-99a6291620e8-tmp\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.433288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.433028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-installation-pull-secrets\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.433450 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.433424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78ba881c-b335-41a8-94c9-99a6291620e8-tmp\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.433845 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.433819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-certificates\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.434034 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.434012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99dd8cb4-20eb-4851-8301-015e25dac6a6-ca-trust-extracted\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.434349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.434331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99dd8cb4-20eb-4851-8301-015e25dac6a6-trusted-ca\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.435790 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.435765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-image-registry-private-configuration\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.435951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.435933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99dd8cb4-20eb-4851-8301-015e25dac6a6-installation-pull-secrets\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.436120 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.436099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-registry-tls\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.436185 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.436143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78ba881c-b335-41a8-94c9-99a6291620e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.455974 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.455940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-bound-sa-token\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.456143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.456068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnsl\" (UniqueName: \"kubernetes.io/projected/99dd8cb4-20eb-4851-8301-015e25dac6a6-kube-api-access-6bnsl\") pod \"image-registry-7bf7d87c9f-99m9w\" (UID: \"99dd8cb4-20eb-4851-8301-015e25dac6a6\") " pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.456784 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.456755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfmb\" (UniqueName: \"kubernetes.io/projected/78ba881c-b335-41a8-94c9-99a6291620e8-kube-api-access-gxfmb\") pod \"klusterlet-addon-workmgr-7f868cb49b-62jth\" (UID: \"78ba881c-b335-41a8-94c9-99a6291620e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.543010 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.542968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:13.587511 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.587479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:13.697698 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.697652    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"]
Jul 28 12:36:13.702991 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:13.702960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ba881c_b335_41a8_94c9_99a6291620e8.slice/crio-1398ea0b5e2a4374801579af11fa9d8953c52790e738913b3ce7ede4f35f5a15 WatchSource:0}: Error finding container 1398ea0b5e2a4374801579af11fa9d8953c52790e738913b3ce7ede4f35f5a15: Status 404 returned error can't find the container with id 1398ea0b5e2a4374801579af11fa9d8953c52790e738913b3ce7ede4f35f5a15
Jul 28 12:36:13.704082 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.704054    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-66s5m"
Jul 28 12:36:13.738801 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:13.738768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf7d87c9f-99m9w"]
Jul 28 12:36:14.164924 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:14.164817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w" event={"ID":"99dd8cb4-20eb-4851-8301-015e25dac6a6","Type":"ContainerStarted","Data":"fc3732da7d96f196377f1c6562b7bcaa030500f46a92192a36149d3890bd8fb3"}
Jul 28 12:36:14.164924 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:14.164900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w" event={"ID":"99dd8cb4-20eb-4851-8301-015e25dac6a6","Type":"ContainerStarted","Data":"00dda2d4ebfc11958c0e4bc1d3c3532c2e1aac76db2675dfafe26e526d03ce43"}
Jul 28 12:36:14.165229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:14.164932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:14.166129 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:14.166107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth" event={"ID":"78ba881c-b335-41a8-94c9-99a6291620e8","Type":"ContainerStarted","Data":"1398ea0b5e2a4374801579af11fa9d8953c52790e738913b3ce7ede4f35f5a15"}
Jul 28 12:36:14.204714 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:14.204661    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w" podStartSLOduration=1.204643564 podStartE2EDuration="1.204643564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:14.203240027 +0000 UTC m=+64.037985198" watchObservedRunningTime="2026-07-28 12:36:14.204643564 +0000 UTC m=+64.039388737"
Jul 28 12:36:15.454765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.454727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:36:15.455234 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.454844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:36:15.457965 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.457943    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:36:15.458819 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.458802    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:36:15.467508 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.467484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/785026c7-2c37-44f5-82ae-cc32f614b0d0-original-pull-secret\") pod \"global-pull-secret-syncer-llr2w\" (UID: \"785026c7-2c37-44f5-82ae-cc32f614b0d0\") " pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:36:15.467593 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.467482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291e2747-1e3c-46df-bd4e-91631ff384f1-metrics-certs\") pod \"network-metrics-daemon-86zqw\" (UID: \"291e2747-1e3c-46df-bd4e-91631ff384f1\") " pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:36:15.555876 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.555820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:36:15.559162 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.559134    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:36:15.569192 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.569165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:36:15.579152 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.579120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6gz\" (UniqueName: \"kubernetes.io/projected/28edcfdd-cbed-4aad-b0fb-d2d1677c1adc-kube-api-access-mc6gz\") pod \"network-check-target-ksczq\" (UID: \"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc\") " pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:36:15.674330 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.674291    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z4tgw\""
Jul 28 12:36:15.679703 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.679670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-llr2w"
Jul 28 12:36:15.681799 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.681776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:36:15.688104 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.688081    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kxg2r\""
Jul 28 12:36:15.695966 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.695937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86zqw"
Jul 28 12:36:15.838659 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.838600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ksczq"]
Jul 28 12:36:15.856567 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.856535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-llr2w"]
Jul 28 12:36:15.859465 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:15.859440    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785026c7_2c37_44f5_82ae_cc32f614b0d0.slice/crio-9d276b3597696e36c1046a695ac4e38d5808f10c546c15012b74e8c2830b4d9f WatchSource:0}: Error finding container 9d276b3597696e36c1046a695ac4e38d5808f10c546c15012b74e8c2830b4d9f: Status 404 returned error can't find the container with id 9d276b3597696e36c1046a695ac4e38d5808f10c546c15012b74e8c2830b4d9f
Jul 28 12:36:15.872740 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:15.872653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86zqw"]
Jul 28 12:36:15.875206 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:15.875179    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291e2747_1e3c_46df_bd4e_91631ff384f1.slice/crio-597cc5c615695b01c68fcaf369acfc05f7fde5df2531c83025f49c44d6766a04 WatchSource:0}: Error finding container 597cc5c615695b01c68fcaf369acfc05f7fde5df2531c83025f49c44d6766a04: Status 404 returned error can't find the container with id 597cc5c615695b01c68fcaf369acfc05f7fde5df2531c83025f49c44d6766a04
Jul 28 12:36:16.174384 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:16.174295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86zqw" event={"ID":"291e2747-1e3c-46df-bd4e-91631ff384f1","Type":"ContainerStarted","Data":"597cc5c615695b01c68fcaf369acfc05f7fde5df2531c83025f49c44d6766a04"}
Jul 28 12:36:16.175383 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:16.175359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ksczq" event={"ID":"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc","Type":"ContainerStarted","Data":"97340e2e302a6c6a90291583f7b199e60d4c53aaa8945851ca35bc2c40fd65bf"}
Jul 28 12:36:16.176399 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:16.176377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-llr2w" event={"ID":"785026c7-2c37-44f5-82ae-cc32f614b0d0","Type":"ContainerStarted","Data":"9d276b3597696e36c1046a695ac4e38d5808f10c546c15012b74e8c2830b4d9f"}
Jul 28 12:36:16.966789 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:16.966678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:16.971113 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:16.971039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f933002c-75ff-463e-90c2-e42816673f7f-metrics-tls\") pod \"dns-default-nkh8w\" (UID: \"f933002c-75ff-463e-90c2-e42816673f7f\") " pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:17.174213 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.174037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l7wd\""
Jul 28 12:36:17.181342 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.181309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:17.183682 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.183142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86zqw" event={"ID":"291e2747-1e3c-46df-bd4e-91631ff384f1","Type":"ContainerStarted","Data":"2b8afe053179bbe5bcc74a7a013f440efce25972f19eb0fda65dbd9cbae6751a"}
Jul 28 12:36:17.183682 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.183207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86zqw" event={"ID":"291e2747-1e3c-46df-bd4e-91631ff384f1","Type":"ContainerStarted","Data":"238c30de4c9f8ff38552f295ba47f262b1c90376d5eb14c8788c970608fc26ab"}
Jul 28 12:36:17.203324 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.203214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-86zqw" podStartSLOduration=65.23096795 podStartE2EDuration="1m6.203193584s" totalImagesPullingTime="972.225634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:17.201188429 +0000 UTC m=+67.035933603" watchObservedRunningTime="2026-07-28 12:36:17.203193584 +0000 UTC m=+67.037938750"
Jul 28 12:36:17.347275 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:17.347196    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nkh8w"]
Jul 28 12:36:17.350065 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:17.350030    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf933002c_75ff_463e_90c2_e42816673f7f.slice/crio-19ebf209a75e29a6513b43716f6af191f6bb2deb0356b4f184e04a0f30c6d0d4 WatchSource:0}: Error finding container 19ebf209a75e29a6513b43716f6af191f6bb2deb0356b4f184e04a0f30c6d0d4: Status 404 returned error can't find the container with id 19ebf209a75e29a6513b43716f6af191f6bb2deb0356b4f184e04a0f30c6d0d4
Jul 28 12:36:18.188935 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:18.188897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkh8w" event={"ID":"f933002c-75ff-463e-90c2-e42816673f7f","Type":"ContainerStarted","Data":"19ebf209a75e29a6513b43716f6af191f6bb2deb0356b4f184e04a0f30c6d0d4"}
Jul 28 12:36:19.387568 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:19.387525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:19.388229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:19.388207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0a2c52-afb9-4f06-a3a9-5c5416d72895-service-ca-bundle\") pod \"router-default-6f7786d8cf-rm5kx\" (UID: \"9b0a2c52-afb9-4f06-a3a9-5c5416d72895\") " pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:19.566358 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:19.566327    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-86rvp\""
Jul 28 12:36:19.572954 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:19.572921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:21.299166 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.299032    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f7786d8cf-rm5kx"]
Jul 28 12:36:21.303552 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:21.303515    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0a2c52_afb9_4f06_a3a9_5c5416d72895.slice/crio-15a702e89ad47e403f093d444d0119bbfdb6399030ae489f4dc0e27bcff5a1d1 WatchSource:0}: Error finding container 15a702e89ad47e403f093d444d0119bbfdb6399030ae489f4dc0e27bcff5a1d1: Status 404 returned error can't find the container with id 15a702e89ad47e403f093d444d0119bbfdb6399030ae489f4dc0e27bcff5a1d1
Jul 28 12:36:21.892393 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.892329    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"]
Jul 28 12:36:21.894818 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.894792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:21.899234 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.899010    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 12:36:21.899234 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.899011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-25nsm\""
Jul 28 12:36:21.921209 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:21.921126    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"]
Jul 28 12:36:22.009761 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.009671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-tssq4\" (UID: \"5f840ea5-ed93-4a1f-879f-062223498332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:22.111244 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.111097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-tssq4\" (UID: \"5f840ea5-ed93-4a1f-879f-062223498332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:22.111244 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:22.111205    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 12:36:22.111468 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:22.111303    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates podName:5f840ea5-ed93-4a1f-879f-062223498332 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:22.611283889 +0000 UTC m=+72.446029046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-tssq4" (UID: "5f840ea5-ed93-4a1f-879f-062223498332") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 12:36:22.213825 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.213775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ksczq" event={"ID":"28edcfdd-cbed-4aad-b0fb-d2d1677c1adc","Type":"ContainerStarted","Data":"2e57dc2c21022961e483058832a10018487019dbed08f00643f3d738082e04df"}
Jul 28 12:36:22.214031 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.213883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:36:22.216324 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.216288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-llr2w" event={"ID":"785026c7-2c37-44f5-82ae-cc32f614b0d0","Type":"ContainerStarted","Data":"c6643b70bf2a7e09d974f20f094ca32ced439ab8a3a23df0572f58bf1ee9ee06"}
Jul 28 12:36:22.218775 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.218743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkh8w" event={"ID":"f933002c-75ff-463e-90c2-e42816673f7f","Type":"ContainerStarted","Data":"21945bdfc37277ef4afda098b9a40f8b292886f8754b3edb6da6e99007432ed1"}
Jul 28 12:36:22.218938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.218804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nkh8w" event={"ID":"f933002c-75ff-463e-90c2-e42816673f7f","Type":"ContainerStarted","Data":"72b50d04032426b5b5f1661e9228eddfe11229d25798de7ba040ff8b554527a2"}
Jul 28 12:36:22.218938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.218854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:22.221023 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.220988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx" event={"ID":"9b0a2c52-afb9-4f06-a3a9-5c5416d72895","Type":"ContainerStarted","Data":"ae1ef359e64d29418b871d491d315a8ddb9820ec59c8c2e58cb86b0367b88917"}
Jul 28 12:36:22.221130 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.221032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx" event={"ID":"9b0a2c52-afb9-4f06-a3a9-5c5416d72895","Type":"ContainerStarted","Data":"15a702e89ad47e403f093d444d0119bbfdb6399030ae489f4dc0e27bcff5a1d1"}
Jul 28 12:36:22.242182 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.241908    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ksczq" podStartSLOduration=65.975432253 podStartE2EDuration="1m11.241884241s" totalImagesPullingTime="5.266451988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:22.240230204 +0000 UTC m=+72.074975379" watchObservedRunningTime="2026-07-28 12:36:22.241884241 +0000 UTC m=+72.076629417"
Jul 28 12:36:22.267102 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.267043    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-llr2w" podStartSLOduration=66.005442675 podStartE2EDuration="1m11.267021244s" totalImagesPullingTime="5.261578569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:22.265579102 +0000 UTC m=+72.100324281" watchObservedRunningTime="2026-07-28 12:36:22.267021244 +0000 UTC m=+72.101766437"
Jul 28 12:36:22.290471 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.290414    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx" podStartSLOduration=35.290390734 podStartE2EDuration="35.290390734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:22.289262244 +0000 UTC m=+72.124007420" watchObservedRunningTime="2026-07-28 12:36:22.290390734 +0000 UTC m=+72.125135909"
Jul 28 12:36:22.312320 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.312240    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nkh8w" podStartSLOduration=33.553496304 podStartE2EDuration="37.312220436s" totalImagesPullingTime="3.758724132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:22.311193711 +0000 UTC m=+72.145938911" watchObservedRunningTime="2026-07-28 12:36:22.312220436 +0000 UTC m=+72.146965611"
Jul 28 12:36:22.574017 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.573928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:22.578381 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.578333    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:22.615561 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.615518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-tssq4\" (UID: \"5f840ea5-ed93-4a1f-879f-062223498332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:22.618991 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.618957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f840ea5-ed93-4a1f-879f-062223498332-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-tssq4\" (UID: \"5f840ea5-ed93-4a1f-879f-062223498332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:22.648183 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.648144    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:36:22.656917 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.656857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.660413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.660303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:36:22.660413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.660326    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:36:22.660413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.660313    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:36:22.666963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.660936    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zh7zq\""
Jul 28 12:36:22.666963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.661903    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:36:22.666963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.661937    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:36:22.666963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.662671    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:36:22.666963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.662721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:36:22.671013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.670946    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:36:22.716832 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.716786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.716832 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.716829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.717099 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.716888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.717099 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.716953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.717099 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.717038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.717252 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.717106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfcv\" (UniqueName: \"kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.806832 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.806796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:22.818031 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.817996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818031 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818281 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818281 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818379 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818379 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfcv\" (UniqueName: \"kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.818893 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.818817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.819254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.819178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.819356 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.819306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.821198 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.821173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.821836 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.821814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.831061 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.830983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfcv\" (UniqueName: \"kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv\") pod \"console-5cb4d5dfb8-lb86c\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") " pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:22.977577 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:22.977540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:23.005274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:23.004938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"]
Jul 28 12:36:23.008275 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:23.008233    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f840ea5_ed93_4a1f_879f_062223498332.slice/crio-71bdf201ddd9804f3804a82660fb9f597cf87561e2a08a63d107ddb09bb3a436 WatchSource:0}: Error finding container 71bdf201ddd9804f3804a82660fb9f597cf87561e2a08a63d107ddb09bb3a436: Status 404 returned error can't find the container with id 71bdf201ddd9804f3804a82660fb9f597cf87561e2a08a63d107ddb09bb3a436
Jul 28 12:36:23.154121 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:23.154076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:36:23.226543 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:23.226497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4" event={"ID":"5f840ea5-ed93-4a1f-879f-062223498332","Type":"ContainerStarted","Data":"71bdf201ddd9804f3804a82660fb9f597cf87561e2a08a63d107ddb09bb3a436"}
Jul 28 12:36:23.227288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:23.226913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:23.228269 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:23.228251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f7786d8cf-rm5kx"
Jul 28 12:36:24.260436 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:24.260392    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5ed74f_3761_45b8_b5c9_f4d09c9d3f49.slice/crio-5794953ad5a31ebe7b74bb1443e0f05acf0197cd5610cf36fdcf6291c66a5c9a WatchSource:0}: Error finding container 5794953ad5a31ebe7b74bb1443e0f05acf0197cd5610cf36fdcf6291c66a5c9a: Status 404 returned error can't find the container with id 5794953ad5a31ebe7b74bb1443e0f05acf0197cd5610cf36fdcf6291c66a5c9a
Jul 28 12:36:24.533380 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:24.533281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:24.536300 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:24.536266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36724ce5-e4fa-446d-88de-80d9d42b271d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qkxw\" (UID: \"36724ce5-e4fa-446d-88de-80d9d42b271d\") " pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:24.728311 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:24.728272    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nrjs7\""
Jul 28 12:36:24.735222 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:24.735185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qkxw"
Jul 28 12:36:25.096174 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.096018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qkxw"]
Jul 28 12:36:25.195089 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:25.195050    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36724ce5_e4fa_446d_88de_80d9d42b271d.slice/crio-d3d7d4aca41e13c3584eca9616cc43abe209b46d008f0aedcb94c1c89f56a614 WatchSource:0}: Error finding container d3d7d4aca41e13c3584eca9616cc43abe209b46d008f0aedcb94c1c89f56a614: Status 404 returned error can't find the container with id d3d7d4aca41e13c3584eca9616cc43abe209b46d008f0aedcb94c1c89f56a614
Jul 28 12:36:25.234891 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.234842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb4d5dfb8-lb86c" event={"ID":"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49","Type":"ContainerStarted","Data":"5794953ad5a31ebe7b74bb1443e0f05acf0197cd5610cf36fdcf6291c66a5c9a"}
Jul 28 12:36:25.236129 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.236086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qkxw" event={"ID":"36724ce5-e4fa-446d-88de-80d9d42b271d","Type":"ContainerStarted","Data":"d3d7d4aca41e13c3584eca9616cc43abe209b46d008f0aedcb94c1c89f56a614"}
Jul 28 12:36:25.237768 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.237731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth" event={"ID":"78ba881c-b335-41a8-94c9-99a6291620e8","Type":"ContainerStarted","Data":"a4fe1558f0710e7f69436882c3be2484423e9d80d72ad2e3b1ab0d81a3532fc6"}
Jul 28 12:36:25.238048 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.238018    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:25.240202 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.240130    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth"
Jul 28 12:36:25.266740 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:25.266679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f868cb49b-62jth" podStartSLOduration=1.627293409 podStartE2EDuration="12.266659996s" totalImagesPullingTime="10.639366587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.266518914 +0000 UTC m=+75.101264091" watchObservedRunningTime="2026-07-28 12:36:25.266659996 +0000 UTC m=+75.101405171"
Jul 28 12:36:26.243509 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:26.243462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4" event={"ID":"5f840ea5-ed93-4a1f-879f-062223498332","Type":"ContainerStarted","Data":"c28c89a3d7b7cba1df585ecce8c2604b00dfa769a3882a5357ba51732cb6f936"}
Jul 28 12:36:26.269594 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:26.269538    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4" podStartSLOduration=3.070132641 podStartE2EDuration="5.269521102s" totalImagesPullingTime="2.199388461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:26.268158859 +0000 UTC m=+76.102904058" watchObservedRunningTime="2026-07-28 12:36:26.269521102 +0000 UTC m=+76.104266277"
Jul 28 12:36:27.248365 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:27.248331    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:27.255536 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:27.255414    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-tssq4"
Jul 28 12:36:28.255496 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:28.255408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb4d5dfb8-lb86c" event={"ID":"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49","Type":"ContainerStarted","Data":"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"}
Jul 28 12:36:28.290023 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:28.289957    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cb4d5dfb8-lb86c" podStartSLOduration=2.545938513 podStartE2EDuration="6.289932477s" totalImagesPullingTime="3.743993964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:28.288237274 +0000 UTC m=+78.122982473" watchObservedRunningTime="2026-07-28 12:36:28.289932477 +0000 UTC m=+78.124677651"
Jul 28 12:36:31.269347 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:31.269302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qkxw" event={"ID":"36724ce5-e4fa-446d-88de-80d9d42b271d","Type":"ContainerStarted","Data":"dcb5eb24ca7302f5cceda2f1335b5cb949a0bff94748c90a2bc67cde43b62466"}
Jul 28 12:36:32.230135 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:32.229999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nkh8w"
Jul 28 12:36:32.978628 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:32.978593    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:32.979086 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:32.978667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:32.980267 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:32.980233    2560 patch_prober.go:28] interesting pod/console-5cb4d5dfb8-lb86c container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused" start-of-body=
Jul 28 12:36:32.980393 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:32.980286    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5cb4d5dfb8-lb86c" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerName="console" probeResult="failure" output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused"
Jul 28 12:36:33.593620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:33.593585    2560 patch_prober.go:28] interesting pod/image-registry-7bf7d87c9f-99m9w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 12:36:33.593840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:33.593639    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w" podUID="99dd8cb4-20eb-4851-8301-015e25dac6a6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:36:35.175128 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:35.175096    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bf7d87c9f-99m9w"
Jul 28 12:36:35.288333 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:35.288287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qkxw" event={"ID":"36724ce5-e4fa-446d-88de-80d9d42b271d","Type":"ContainerStarted","Data":"420b096f1d50bfcfad099c74265b9188c49ea162b8cec9d4c4becae0776825b5"}
Jul 28 12:36:38.304681 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:38.304595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qkxw" event={"ID":"36724ce5-e4fa-446d-88de-80d9d42b271d","Type":"ContainerStarted","Data":"9fabe40c9ac4e80d8c2f3c74d29aa782d6e54ddc9f567be48830620fd3e66639"}
Jul 28 12:36:38.325272 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:38.325214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2qkxw" podStartSLOduration=33.597595921 podStartE2EDuration="46.325195074s" totalImagesPullingTime="12.727599153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:38.324082436 +0000 UTC m=+88.158827626" watchObservedRunningTime="2026-07-28 12:36:38.325195074 +0000 UTC m=+88.159940247"
Jul 28 12:36:42.982565 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:42.982533    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:42.986665 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:42.986631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:36:45.971838 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:45.971801    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:36:47.377554 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.377510    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"]
Jul 28 12:36:47.382305 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.382278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.385150 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.385120    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 12:36:47.385496 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.385474    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5cjds\""
Jul 28 12:36:47.385606 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.385525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:36:47.386911 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.386741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 12:36:47.386911 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.386744    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:36:47.386911 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.386807    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:36:47.386911 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.386814    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:36:47.394458 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.394424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"]
Jul 28 12:36:47.400759 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.400722    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-52wd5"]
Jul 28 12:36:47.404475 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.404448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.409163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.409137    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:36:47.409404 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.409167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r62cm\""
Jul 28 12:36:47.409545 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.409183    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:36:47.409657 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.409220    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:36:47.531431 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-metrics-client-ca\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92w4\" (UniqueName: \"kubernetes.io/projected/a383215a-283d-4447-87d4-72ae1f827fb1-kube-api-access-c92w4\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-root\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-wtmp\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-textfile\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-tls\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a383215a-283d-4447-87d4-72ae1f827fb1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.531852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.532090 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-sys\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.532090 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.531924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47t4\" (UniqueName: \"kubernetes.io/projected/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-kube-api-access-t47t4\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633405 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-sys\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633405 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47t4\" (UniqueName: \"kubernetes.io/projected/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-kube-api-access-t47t4\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-sys\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-metrics-client-ca\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92w4\" (UniqueName: \"kubernetes.io/projected/a383215a-283d-4447-87d4-72ae1f827fb1-kube-api-access-c92w4\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-root\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-wtmp\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-textfile\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.633620 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-root\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-wtmp\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-tls\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.633973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-textfile\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634080 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:47.634040    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 12:36:47.634366 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.634108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-metrics-client-ca\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634366 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:36:47.634125    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls podName:a383215a-283d-4447-87d4-72ae1f827fb1 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:48.134101792 +0000 UTC m=+97.968846947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-zm4tt" (UID: "a383215a-283d-4447-87d4-72ae1f827fb1") : secret "kube-state-metrics-tls" not found
Jul 28 12:36:47.634480 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.634369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.634480 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.634464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.634588 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.634506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a383215a-283d-4447-87d4-72ae1f827fb1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.634588 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.634534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.635103 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.635076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a383215a-283d-4447-87d4-72ae1f827fb1-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.635279 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.635194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a383215a-283d-4447-87d4-72ae1f827fb1-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.635441 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.635405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.636654 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.636625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-tls\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.642823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.642784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.643743 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.643720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.647178 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.647149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47t4\" (UniqueName: \"kubernetes.io/projected/f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e-kube-api-access-t47t4\") pod \"node-exporter-52wd5\" (UID: \"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e\") " pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.647326 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.647296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92w4\" (UniqueName: \"kubernetes.io/projected/a383215a-283d-4447-87d4-72ae1f827fb1-kube-api-access-c92w4\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:47.719834 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:47.719793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-52wd5"
Jul 28 12:36:47.733182 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:47.733150    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fbbfb3_1de8_42fe_ba7a_5790c25faf9e.slice/crio-3afb60f81d4dc17d86a705d2d683f493d02b0cb097e4cfdc60321adfbdec04ce WatchSource:0}: Error finding container 3afb60f81d4dc17d86a705d2d683f493d02b0cb097e4cfdc60321adfbdec04ce: Status 404 returned error can't find the container with id 3afb60f81d4dc17d86a705d2d683f493d02b0cb097e4cfdc60321adfbdec04ce
Jul 28 12:36:48.138748 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:48.138703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:48.141110 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:48.141083    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a383215a-283d-4447-87d4-72ae1f827fb1-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-zm4tt\" (UID: \"a383215a-283d-4447-87d4-72ae1f827fb1\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:48.293518 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:48.293476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"
Jul 28 12:36:48.342768 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:48.342729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-52wd5" event={"ID":"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e","Type":"ContainerStarted","Data":"3afb60f81d4dc17d86a705d2d683f493d02b0cb097e4cfdc60321adfbdec04ce"}
Jul 28 12:36:48.447320 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:48.447287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt"]
Jul 28 12:36:48.450334 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:48.450307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda383215a_283d_4447_87d4_72ae1f827fb1.slice/crio-70fb61f2724dfff602353a9957ea357ad40ccca43a0925f4a0995132cbb0efb9 WatchSource:0}: Error finding container 70fb61f2724dfff602353a9957ea357ad40ccca43a0925f4a0995132cbb0efb9: Status 404 returned error can't find the container with id 70fb61f2724dfff602353a9957ea357ad40ccca43a0925f4a0995132cbb0efb9
Jul 28 12:36:49.348834 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.348792    2560 generic.go:356] "Generic (PLEG): container finished" podID="f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e" containerID="a090a6ddda0d4b3b02422d01e514a3de0c0b5fd5ded7552fbf0dd6730e61d9c4" exitCode=0
Jul 28 12:36:49.349050 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.348904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-52wd5" event={"ID":"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e","Type":"ContainerDied","Data":"a090a6ddda0d4b3b02422d01e514a3de0c0b5fd5ded7552fbf0dd6730e61d9c4"}
Jul 28 12:36:49.350395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.350327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt" event={"ID":"a383215a-283d-4447-87d4-72ae1f827fb1","Type":"ContainerStarted","Data":"70fb61f2724dfff602353a9957ea357ad40ccca43a0925f4a0995132cbb0efb9"}
Jul 28 12:36:49.425308 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.425260    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ff9454757-5ntl5"]
Jul 28 12:36:49.429381 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.429358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.433035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.432855    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 12:36:49.433035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.432892    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 12:36:49.433035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.432925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 12:36:49.433035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.432940    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 12:36:49.433377 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.433061    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-as3jv6h2sgsuc\""
Jul 28 12:36:49.433377 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.433169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5s984\""
Jul 28 12:36:49.433377 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.433246    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 12:36:49.444030 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.443997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff9454757-5ntl5"]
Jul 28 12:36:49.551414 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551414 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef23623-bf03-4949-a3c7-29a490173793-metrics-client-ca\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqjg\" (UniqueName: \"kubernetes.io/projected/5ef23623-bf03-4949-a3c7-29a490173793-kube-api-access-lhqjg\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.551938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.551749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-grpc-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.652917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.652984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqjg\" (UniqueName: \"kubernetes.io/projected/5ef23623-bf03-4949-a3c7-29a490173793-kube-api-access-lhqjg\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-grpc-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.653282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef23623-bf03-4949-a3c7-29a490173793-metrics-client-ca\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.654004 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.653974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef23623-bf03-4949-a3c7-29a490173793-metrics-client-ca\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656488 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656852 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.656993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.656950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ef23623-bf03-4949-a3c7-29a490173793-secret-grpc-tls\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.663433 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.663400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqjg\" (UniqueName: \"kubernetes.io/projected/5ef23623-bf03-4949-a3c7-29a490173793-kube-api-access-lhqjg\") pod \"thanos-querier-5ff9454757-5ntl5\" (UID: \"5ef23623-bf03-4949-a3c7-29a490173793\") " pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:49.761037 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:49.760998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:50.047734 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.047703    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff9454757-5ntl5"]
Jul 28 12:36:50.050434 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:50.050394    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef23623_bf03_4949_a3c7_29a490173793.slice/crio-1fa0b56297e79899f208c47ed8c2afb020ab64e1c2be3dfc4d47802a252a73ff WatchSource:0}: Error finding container 1fa0b56297e79899f208c47ed8c2afb020ab64e1c2be3dfc4d47802a252a73ff: Status 404 returned error can't find the container with id 1fa0b56297e79899f208c47ed8c2afb020ab64e1c2be3dfc4d47802a252a73ff
Jul 28 12:36:50.356726 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.356667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"1fa0b56297e79899f208c47ed8c2afb020ab64e1c2be3dfc4d47802a252a73ff"}
Jul 28 12:36:50.359262 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.359227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-52wd5" event={"ID":"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e","Type":"ContainerStarted","Data":"e647b05d978870829c550e038a0c72cae1ec841a492e55fc03824f45b6cd3d4a"}
Jul 28 12:36:50.359434 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.359289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-52wd5" event={"ID":"f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e","Type":"ContainerStarted","Data":"577d3036bba0aa0c8d9bb2f7fa419e87cbb9ea18370ac5365cdd0df8637ddaea"}
Jul 28 12:36:50.361577 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.361540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt" event={"ID":"a383215a-283d-4447-87d4-72ae1f827fb1","Type":"ContainerStarted","Data":"2391ce93dd5f9c086839ee09678da2f41b6f12a3ab698723584f417ba4437e17"}
Jul 28 12:36:50.361708 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.361605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt" event={"ID":"a383215a-283d-4447-87d4-72ae1f827fb1","Type":"ContainerStarted","Data":"d17103fd88dbd809fefbcf4439f6b44d25ca15a97a618f04acf0b0213be79199"}
Jul 28 12:36:50.361708 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.361633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt" event={"ID":"a383215a-283d-4447-87d4-72ae1f827fb1","Type":"ContainerStarted","Data":"65477db07498cf3dcfa4640ac61c82fff87bcbaa064bb8eb552e98e133f5e3ca"}
Jul 28 12:36:50.381977 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.381912    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-52wd5" podStartSLOduration=1.949806452 podStartE2EDuration="3.381890226s" totalImagesPullingTime="1.366250806s" totalInitContainerRuntime="65.832968ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:50.38095459 +0000 UTC m=+100.215699791" watchObservedRunningTime="2026-07-28 12:36:50.381890226 +0000 UTC m=+100.216635403"
Jul 28 12:36:50.401612 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:50.401547    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-zm4tt" podStartSLOduration=1.906379595 podStartE2EDuration="3.401525443s" totalImagesPullingTime="1.495145848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:50.401004871 +0000 UTC m=+100.235750068" watchObservedRunningTime="2026-07-28 12:36:50.401525443 +0000 UTC m=+100.236270620"
Jul 28 12:36:52.130100 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.130061    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:36:52.133840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.133814    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.147140 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.147048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:36:52.153544 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.153517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:36:52.172706 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.172910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.172910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.172910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.172910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.173078 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.173078 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.172979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdmj\" (UniqueName: \"kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274082 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274082 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274331 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274331 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274331 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274482 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdmj\" (UniqueName: \"kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274482 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.274986 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.274953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.275112 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.275052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.275112 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.275098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.275297 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.275280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.277189 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.277164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.277281 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.277188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.283613 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.283583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdmj\" (UniqueName: \"kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj\") pod \"console-84c7994b69-wn9qg\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") " pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.372048 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.372015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"dbfcd08c38b30cfc2ea5fb222ab1d08e2deb69f325271a8f618abe3df4bf5134"}
Jul 28 12:36:52.372208 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.372090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"29317f17025347e7a96037a765c1ce3a3f590b7cd8bb7817c58df97ffdae12d2"}
Jul 28 12:36:52.372208 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.372127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"833a4a0639178d605ea6a21d3d3c4f932e95ecb1fd23172c79202acbb9c06990"}
Jul 28 12:36:52.451759 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.451718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:36:52.605845 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:52.605811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:36:52.609324 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:36:52.609296    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0534da8_7343_4475_8f66_05c84f155604.slice/crio-9cbabfa5b999c584faa4d6ebfdcc69e90ab1aebf8bbb879df61dea9c8afb5659 WatchSource:0}: Error finding container 9cbabfa5b999c584faa4d6ebfdcc69e90ab1aebf8bbb879df61dea9c8afb5659: Status 404 returned error can't find the container with id 9cbabfa5b999c584faa4d6ebfdcc69e90ab1aebf8bbb879df61dea9c8afb5659
Jul 28 12:36:53.229581 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:53.229550    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ksczq"
Jul 28 12:36:53.379194 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:53.379153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"0d34406354f9f8f9d321e86950d0ef5dfd8e9e827e0dd0a3a384f2e5f7ee20cb"}
Jul 28 12:36:53.380616 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:53.380587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c7994b69-wn9qg" event={"ID":"b0534da8-7343-4475-8f66-05c84f155604","Type":"ContainerStarted","Data":"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"}
Jul 28 12:36:53.380789 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:53.380631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c7994b69-wn9qg" event={"ID":"b0534da8-7343-4475-8f66-05c84f155604","Type":"ContainerStarted","Data":"9cbabfa5b999c584faa4d6ebfdcc69e90ab1aebf8bbb879df61dea9c8afb5659"}
Jul 28 12:36:53.404273 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:53.404215    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c7994b69-wn9qg" podStartSLOduration=1.404196794 podStartE2EDuration="1.404196794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:53.402715538 +0000 UTC m=+103.237460733" watchObservedRunningTime="2026-07-28 12:36:53.404196794 +0000 UTC m=+103.238941969"
Jul 28 12:36:54.387584 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:54.387542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"b2d573af4f10e9c3cb58b647fe0984ff51fafa7e712de7b5f5875a4ed687b847"}
Jul 28 12:36:54.388085 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:54.387627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" event={"ID":"5ef23623-bf03-4949-a3c7-29a490173793","Type":"ContainerStarted","Data":"8c0b1fc761ef913701758764504154b4ec958e1ccdc280c9ab16c35e4898cd7c"}
Jul 28 12:36:54.419815 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:54.419758    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5" podStartSLOduration=2.444215849 podStartE2EDuration="5.419739002s" totalImagesPullingTime="2.975523153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:54.418045612 +0000 UTC m=+104.252790822" watchObservedRunningTime="2026-07-28 12:36:54.419739002 +0000 UTC m=+104.254484175"
Jul 28 12:36:55.391651 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:55.391616    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:36:57.828058 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:36:57.828021    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:37:01.401697 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:01.401666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ff9454757-5ntl5"
Jul 28 12:37:02.452668 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:02.452626    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:37:09.444491 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:09.444456    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcd4d309-870a-4c7c-b841-bcd5bd5235e5" containerID="4960699e12e0f9201df84b7f62a758f19ffcdbe285dce37d529c36e446b8ce73" exitCode=0
Jul 28 12:37:09.444946 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:09.444524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm" event={"ID":"dcd4d309-870a-4c7c-b841-bcd5bd5235e5","Type":"ContainerDied","Data":"4960699e12e0f9201df84b7f62a758f19ffcdbe285dce37d529c36e446b8ce73"}
Jul 28 12:37:09.444946 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:09.444851    2560 scope.go:117] "RemoveContainer" containerID="4960699e12e0f9201df84b7f62a758f19ffcdbe285dce37d529c36e446b8ce73"
Jul 28 12:37:10.451173 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:10.451139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm" event={"ID":"dcd4d309-870a-4c7c-b841-bcd5bd5235e5","Type":"ContainerStarted","Data":"0ab5da5886c1afa7efc640f278e0756f3615b86594e4f157234c478a67abc774"}
Jul 28 12:37:10.992391 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:10.992321    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cb4d5dfb8-lb86c" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerName="console" containerID="cri-o://ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4" gracePeriod=15
Jul 28 12:37:11.239950 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.239922    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:37:11.350180 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350089    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350180 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350180 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350459 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350459 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350233    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zfcv\" (UniqueName: \"kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350459 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca\") pod \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\" (UID: \"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49\") "
Jul 28 12:37:11.350608 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:11.350702 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350677    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config" (OuterVolumeSpecName: "console-config") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:11.350753 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.350674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:11.352588 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.352550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv" (OuterVolumeSpecName: "kube-api-access-9zfcv") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "kube-api-access-9zfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:11.352588 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.352567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:11.352732 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.352585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" (UID: "9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:11.451780 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451747    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.451780 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451775    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.451780 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451784    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.451780 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451793    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.452301 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451802    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zfcv\" (UniqueName: \"kubernetes.io/projected/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-kube-api-access-9zfcv\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.452301 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.451811    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:11.456332 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.456302    2560 generic.go:356] "Generic (PLEG): container finished" podID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerID="ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4" exitCode=0
Jul 28 12:37:11.456495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.456347    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb4d5dfb8-lb86c"
Jul 28 12:37:11.456495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.456392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb4d5dfb8-lb86c" event={"ID":"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49","Type":"ContainerDied","Data":"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"}
Jul 28 12:37:11.456495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.456450    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb4d5dfb8-lb86c" event={"ID":"9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49","Type":"ContainerDied","Data":"5794953ad5a31ebe7b74bb1443e0f05acf0197cd5610cf36fdcf6291c66a5c9a"}
Jul 28 12:37:11.456495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.456479    2560 scope.go:117] "RemoveContainer" containerID="ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"
Jul 28 12:37:11.465265 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.465246    2560 scope.go:117] "RemoveContainer" containerID="ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"
Jul 28 12:37:11.465582 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:37:11.465558    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4\": container with ID starting with ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4 not found: ID does not exist" containerID="ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"
Jul 28 12:37:11.465658 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.465595    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4"} err="failed to get container status \"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4\": rpc error: code = NotFound desc = could not find container \"ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4\": container with ID starting with ac0ffd44cbbe46c6d87d0e7308606e0614c9a71ce0f1a65daa772b2e8f8cf4b4 not found: ID does not exist"
Jul 28 12:37:11.477537 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.477501    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:37:11.483131 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:11.483100    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cb4d5dfb8-lb86c"]
Jul 28 12:37:12.661063 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:12.661028    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" path="/var/lib/kubelet/pods/9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49/volumes"
Jul 28 12:37:18.131667 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:18.131642    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7786d8cf-rm5kx_9b0a2c52-afb9-4f06-a3a9-5c5416d72895/router/0.log"
Jul 28 12:37:18.145073 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:18.145039    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jznjs_922664ba-bfbd-4fa5-8d4f-6104e46d0567/serve-healthcheck-canary/0.log"
Jul 28 12:37:22.847317 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:22.847252    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84c7994b69-wn9qg" podUID="b0534da8-7343-4475-8f66-05c84f155604" containerName="console" containerID="cri-o://bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574" gracePeriod=15
Jul 28 12:37:23.119917 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.119889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:37:23.255009 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.254973    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qdmj\" (UniqueName: \"kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255367 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255367 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255367 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config\") pod \"b0534da8-7343-4475-8f66-05c84f155604\" (UID: \"b0534da8-7343-4475-8f66-05c84f155604\") "
Jul 28 12:37:23.255534 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255362    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:23.255592 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255555    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.255691 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:23.255763 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255679    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca" (OuterVolumeSpecName: "service-ca") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:23.255763 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.255667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config" (OuterVolumeSpecName: "console-config") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:23.257376 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.257344    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:23.257376 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.257367    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:23.257519 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.257350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj" (OuterVolumeSpecName: "kube-api-access-9qdmj") pod "b0534da8-7343-4475-8f66-05c84f155604" (UID: "b0534da8-7343-4475-8f66-05c84f155604"). InnerVolumeSpecName "kube-api-access-9qdmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356664    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356696    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0534da8-7343-4475-8f66-05c84f155604-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356706    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdmj\" (UniqueName: \"kubernetes.io/projected/b0534da8-7343-4475-8f66-05c84f155604-kube-api-access-9qdmj\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356715    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-trusted-ca-bundle\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356724    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.356760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.356732    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0534da8-7343-4475-8f66-05c84f155604-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:37:23.502560 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.502525    2560 generic.go:356] "Generic (PLEG): container finished" podID="b0534da8-7343-4475-8f66-05c84f155604" containerID="bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574" exitCode=0
Jul 28 12:37:23.502772 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.502581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c7994b69-wn9qg"
Jul 28 12:37:23.502772 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.502614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c7994b69-wn9qg" event={"ID":"b0534da8-7343-4475-8f66-05c84f155604","Type":"ContainerDied","Data":"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"}
Jul 28 12:37:23.502772 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.502668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c7994b69-wn9qg" event={"ID":"b0534da8-7343-4475-8f66-05c84f155604","Type":"ContainerDied","Data":"9cbabfa5b999c584faa4d6ebfdcc69e90ab1aebf8bbb879df61dea9c8afb5659"}
Jul 28 12:37:23.502772 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.502691    2560 scope.go:117] "RemoveContainer" containerID="bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"
Jul 28 12:37:23.511845 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.511824    2560 scope.go:117] "RemoveContainer" containerID="bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"
Jul 28 12:37:23.512218 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:37:23.512195    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574\": container with ID starting with bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574 not found: ID does not exist" containerID="bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"
Jul 28 12:37:23.512275 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.512227    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574"} err="failed to get container status \"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574\": rpc error: code = NotFound desc = could not find container \"bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574\": container with ID starting with bc900e0e7111e5d6e9ea77153c2439a46b9475aca571382be4b2f457ef27a574 not found: ID does not exist"
Jul 28 12:37:23.523506 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.523473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:37:23.530425 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:23.530387    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c7994b69-wn9qg"]
Jul 28 12:37:24.664397 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:37:24.664352    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0534da8-7343-4475-8f66-05c84f155604" path="/var/lib/kubelet/pods/b0534da8-7343-4475-8f66-05c84f155604/volumes"
Jul 28 12:40:10.527699 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:40:10.527667    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:40:10.532243 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:40:10.532216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:40:10.538285 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:40:10.538249    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:42:00.672933 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.672898    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857b7978db-lz5xw"]
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673208    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerName="console"
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673221    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerName="console"
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673242    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0534da8-7343-4475-8f66-05c84f155604" containerName="console"
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673248    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0534da8-7343-4475-8f66-05c84f155604" containerName="console"
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673293    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5ed74f-3761-45b8-b5c9-f4d09c9d3f49" containerName="console"
Jul 28 12:42:00.673390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.673305    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0534da8-7343-4475-8f66-05c84f155604" containerName="console"
Jul 28 12:42:00.676393 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.676367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.680836 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.680810    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:42:00.681515 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.681491    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:42:00.681515 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.681519    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:42:00.682219 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.682199    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:42:00.682338 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.682249    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:42:00.682562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.682534    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:42:00.682682 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.682649    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:42:00.682682 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.682661    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zh7zq\""
Jul 28 12:42:00.689771 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.689747    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:42:00.709390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.709348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857b7978db-lz5xw"]
Jul 28 12:42:00.742412 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlz6\" (UniqueName: \"kubernetes.io/projected/509fbcd8-7511-418e-83a0-0f652d196cc1-kube-api-access-vdlz6\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-trusted-ca-bundle\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-oauth-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-console-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-service-ca\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.742717 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.742656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-oauth-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.843882 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-oauth-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlz6\" (UniqueName: \"kubernetes.io/projected/509fbcd8-7511-418e-83a0-0f652d196cc1-kube-api-access-vdlz6\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-trusted-ca-bundle\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-oauth-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-console-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.843983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-service-ca\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.844007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844819 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.844788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-console-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844944 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.844819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-oauth-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844944 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.844827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-service-ca\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.844944 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.844843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509fbcd8-7511-418e-83a0-0f652d196cc1-trusted-ca-bundle\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.846939 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.846918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-serving-cert\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.847069 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.847047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/509fbcd8-7511-418e-83a0-0f652d196cc1-console-oauth-config\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.853893 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.853849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlz6\" (UniqueName: \"kubernetes.io/projected/509fbcd8-7511-418e-83a0-0f652d196cc1-kube-api-access-vdlz6\") pod \"console-857b7978db-lz5xw\" (UID: \"509fbcd8-7511-418e-83a0-0f652d196cc1\") " pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:00.987990 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:00.987953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:01.137296 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:01.137266    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857b7978db-lz5xw"]
Jul 28 12:42:01.140491 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:42:01.140461    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509fbcd8_7511_418e_83a0_0f652d196cc1.slice/crio-dbc01a9fcab9db2286c2e7e315f53128a3cea5da50b2f470edef878486f46ac1 WatchSource:0}: Error finding container dbc01a9fcab9db2286c2e7e315f53128a3cea5da50b2f470edef878486f46ac1: Status 404 returned error can't find the container with id dbc01a9fcab9db2286c2e7e315f53128a3cea5da50b2f470edef878486f46ac1
Jul 28 12:42:01.529319 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:01.529278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857b7978db-lz5xw" event={"ID":"509fbcd8-7511-418e-83a0-0f652d196cc1","Type":"ContainerStarted","Data":"6ffb49094a5c8b522e56a73a09ae970d4fd65f7aa6bc449178542191e75a9fb1"}
Jul 28 12:42:01.529495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:01.529372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857b7978db-lz5xw" event={"ID":"509fbcd8-7511-418e-83a0-0f652d196cc1","Type":"ContainerStarted","Data":"dbc01a9fcab9db2286c2e7e315f53128a3cea5da50b2f470edef878486f46ac1"}
Jul 28 12:42:01.550223 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:01.550167    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857b7978db-lz5xw" podStartSLOduration=1.550149768 podStartE2EDuration="1.550149768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:01.54877911 +0000 UTC m=+411.383524301" watchObservedRunningTime="2026-07-28 12:42:01.550149768 +0000 UTC m=+411.384894942"
Jul 28 12:42:10.988332 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:10.988294    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:10.988736 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:10.988394    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:10.993071 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:10.993046    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:11.570509 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:11.570479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857b7978db-lz5xw"
Jul 28 12:42:56.606902 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.606841    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:42:56.610167 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.610146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.613059 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.613030    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 12:42:56.613510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.613491    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:42:56.614354 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.614326    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-stwbp\""
Jul 28 12:42:56.614465 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.614379    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:42:56.620371 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.620343    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:42:56.643256 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.643215    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-d279j"]
Jul 28 12:42:56.646604 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.646575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.649210 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.649187    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-64wk2\""
Jul 28 12:42:56.649362 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.649189    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 12:42:56.661893 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.661849    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d279j"]
Jul 28 12:42:56.715284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.715246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.715466 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.715338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67fq6\" (UniqueName: \"kubernetes.io/projected/8a51352a-afb7-41d9-a2c2-5910874654b7-kube-api-access-67fq6\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.715466 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.715409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a51352a-afb7-41d9-a2c2-5910874654b7-data\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.715466 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.715430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmkt2\" (UniqueName: \"kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.815932 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.815879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a51352a-afb7-41d9-a2c2-5910874654b7-data\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.816136 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.815944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmkt2\" (UniqueName: \"kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.816136 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.816012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.816136 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.816057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67fq6\" (UniqueName: \"kubernetes.io/projected/8a51352a-afb7-41d9-a2c2-5910874654b7-kube-api-access-67fq6\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.816362 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.816271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a51352a-afb7-41d9-a2c2-5910874654b7-data\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.818558 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.818525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.824911 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.824856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67fq6\" (UniqueName: \"kubernetes.io/projected/8a51352a-afb7-41d9-a2c2-5910874654b7-kube-api-access-67fq6\") pod \"seaweedfs-64568bcd49-d279j\" (UID: \"8a51352a-afb7-41d9-a2c2-5910874654b7\") " pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:56.825317 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.825297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmkt2\" (UniqueName: \"kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2\") pod \"kserve-controller-manager-68bc5447d6-c5n7l\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") " pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.921081 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.920978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:42:56.957471 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:56.957434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:42:57.063604 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:57.063560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:42:57.066048 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:42:57.066008    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5adee6_eb39_4dd4_89f2_1f0395c76866.slice/crio-5f5c24980dd7c500875ef6985a3736e187b86f8853a5ad45a6c1a54a3418d450 WatchSource:0}: Error finding container 5f5c24980dd7c500875ef6985a3736e187b86f8853a5ad45a6c1a54a3418d450: Status 404 returned error can't find the container with id 5f5c24980dd7c500875ef6985a3736e187b86f8853a5ad45a6c1a54a3418d450
Jul 28 12:42:57.067492 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:57.067469    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:42:57.106370 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:57.106336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d279j"]
Jul 28 12:42:57.109231 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:42:57.109202    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a51352a_afb7_41d9_a2c2_5910874654b7.slice/crio-90b394c702e3a181329d6f3846bf253c6edaebe19717d04c01d2ba52e9a4a7a6 WatchSource:0}: Error finding container 90b394c702e3a181329d6f3846bf253c6edaebe19717d04c01d2ba52e9a4a7a6: Status 404 returned error can't find the container with id 90b394c702e3a181329d6f3846bf253c6edaebe19717d04c01d2ba52e9a4a7a6
Jul 28 12:42:57.737687 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:57.737643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" event={"ID":"ac5adee6-eb39-4dd4-89f2-1f0395c76866","Type":"ContainerStarted","Data":"5f5c24980dd7c500875ef6985a3736e187b86f8853a5ad45a6c1a54a3418d450"}
Jul 28 12:42:57.739177 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:42:57.739147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d279j" event={"ID":"8a51352a-afb7-41d9-a2c2-5910874654b7","Type":"ContainerStarted","Data":"90b394c702e3a181329d6f3846bf253c6edaebe19717d04c01d2ba52e9a4a7a6"}
Jul 28 12:43:00.753078 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:00.753040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" event={"ID":"ac5adee6-eb39-4dd4-89f2-1f0395c76866","Type":"ContainerStarted","Data":"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"}
Jul 28 12:43:00.753470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:00.753170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:43:00.776424 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:00.776357    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" podStartSLOduration=1.238186184 podStartE2EDuration="4.776339753s" totalImagesPullingTime="3.538153569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:00.774518496 +0000 UTC m=+470.609263671" watchObservedRunningTime="2026-07-28 12:43:00.776339753 +0000 UTC m=+470.611084927"
Jul 28 12:43:01.758097 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:01.758059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d279j" event={"ID":"8a51352a-afb7-41d9-a2c2-5910874654b7","Type":"ContainerStarted","Data":"c2a02611d8aa236f8bda17ba775f0abd44f16495be339ac308564367b156f238"}
Jul 28 12:43:01.758495 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:01.758133    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:43:01.785767 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:01.785711    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-d279j" podStartSLOduration=2.199654457 podStartE2EDuration="5.78569175s" totalImagesPullingTime="3.586037293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:01.783918349 +0000 UTC m=+471.618663537" watchObservedRunningTime="2026-07-28 12:43:01.78569175 +0000 UTC m=+471.620436925"
Jul 28 12:43:07.764821 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:07.764782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-d279j"
Jul 28 12:43:31.763478 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:31.763445    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:43:39.445355 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.445316    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:43:39.445840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.445509    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" podUID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" containerName="manager" containerID="cri-o://9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae" gracePeriod=10
Jul 28 12:43:39.476604 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.476563    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wjknz"]
Jul 28 12:43:39.479045 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.479028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.491296 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.491263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wjknz"]
Jul 28 12:43:39.570291 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.570252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qc7\" (UniqueName: \"kubernetes.io/projected/b4db8684-0aff-4328-a922-9dc0c9e4b218-kube-api-access-j9qc7\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.570455 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.570299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4db8684-0aff-4328-a922-9dc0c9e4b218-cert\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.671004 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.670858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qc7\" (UniqueName: \"kubernetes.io/projected/b4db8684-0aff-4328-a922-9dc0c9e4b218-kube-api-access-j9qc7\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.671004 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.670919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4db8684-0aff-4328-a922-9dc0c9e4b218-cert\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.673358 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.673323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4db8684-0aff-4328-a922-9dc0c9e4b218-cert\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.679686 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.679658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qc7\" (UniqueName: \"kubernetes.io/projected/b4db8684-0aff-4328-a922-9dc0c9e4b218-kube-api-access-j9qc7\") pod \"kserve-controller-manager-68bc5447d6-wjknz\" (UID: \"b4db8684-0aff-4328-a922-9dc0c9e4b218\") " pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.691252 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.691227    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:43:39.771376 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.771272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmkt2\" (UniqueName: \"kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2\") pod \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") "
Jul 28 12:43:39.771376 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.771350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert\") pod \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\" (UID: \"ac5adee6-eb39-4dd4-89f2-1f0395c76866\") "
Jul 28 12:43:39.773567 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.773531    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2" (OuterVolumeSpecName: "kube-api-access-rmkt2") pod "ac5adee6-eb39-4dd4-89f2-1f0395c76866" (UID: "ac5adee6-eb39-4dd4-89f2-1f0395c76866"). InnerVolumeSpecName "kube-api-access-rmkt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:43:39.773673 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.773604    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert" (OuterVolumeSpecName: "cert") pod "ac5adee6-eb39-4dd4-89f2-1f0395c76866" (UID: "ac5adee6-eb39-4dd4-89f2-1f0395c76866"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:43:39.835042 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.835001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:39.872835 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.872769    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5adee6-eb39-4dd4-89f2-1f0395c76866-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:43:39.872835 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.872806    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmkt2\" (UniqueName: \"kubernetes.io/projected/ac5adee6-eb39-4dd4-89f2-1f0395c76866-kube-api-access-rmkt2\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:43:39.908402 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.908363    2560 generic.go:356] "Generic (PLEG): container finished" podID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" containerID="9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae" exitCode=0
Jul 28 12:43:39.908562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.908410    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l"
Jul 28 12:43:39.908562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.908421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" event={"ID":"ac5adee6-eb39-4dd4-89f2-1f0395c76866","Type":"ContainerDied","Data":"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"}
Jul 28 12:43:39.908562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.908479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-c5n7l" event={"ID":"ac5adee6-eb39-4dd4-89f2-1f0395c76866","Type":"ContainerDied","Data":"5f5c24980dd7c500875ef6985a3736e187b86f8853a5ad45a6c1a54a3418d450"}
Jul 28 12:43:39.908562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.908509    2560 scope.go:117] "RemoveContainer" containerID="9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"
Jul 28 12:43:39.918609 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.918574    2560 scope.go:117] "RemoveContainer" containerID="9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"
Jul 28 12:43:39.919040 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:43:39.919007    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae\": container with ID starting with 9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae not found: ID does not exist" containerID="9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"
Jul 28 12:43:39.919131 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.919054    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae"} err="failed to get container status \"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae\": rpc error: code = NotFound desc = could not find container \"9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae\": container with ID starting with 9b0f630d1d0fa4fae42a8ed155afaf70e5f59ed1c9ebde2c4c686c4d539508ae not found: ID does not exist"
Jul 28 12:43:39.933380 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.933344    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:43:39.940508 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.940469    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-c5n7l"]
Jul 28 12:43:39.969145 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:39.969119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wjknz"]
Jul 28 12:43:39.971647 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:43:39.971618    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db8684_0aff_4328_a922_9dc0c9e4b218.slice/crio-2d889ebcf6ef344725818275bc427ac9eefbb5026ae6896bea527b414c841e24 WatchSource:0}: Error finding container 2d889ebcf6ef344725818275bc427ac9eefbb5026ae6896bea527b414c841e24: Status 404 returned error can't find the container with id 2d889ebcf6ef344725818275bc427ac9eefbb5026ae6896bea527b414c841e24
Jul 28 12:43:40.661174 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:40.661086    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" path="/var/lib/kubelet/pods/ac5adee6-eb39-4dd4-89f2-1f0395c76866/volumes"
Jul 28 12:43:40.914173 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:40.914058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz" event={"ID":"b4db8684-0aff-4328-a922-9dc0c9e4b218","Type":"ContainerStarted","Data":"508f3cd449fbc4d48cce57091a0d87b904178dea55948c1f6324ab2b8ee9e31a"}
Jul 28 12:43:40.914173 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:40.914150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz" event={"ID":"b4db8684-0aff-4328-a922-9dc0c9e4b218","Type":"ContainerStarted","Data":"2d889ebcf6ef344725818275bc427ac9eefbb5026ae6896bea527b414c841e24"}
Jul 28 12:43:40.914393 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:40.914182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:43:40.933810 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:43:40.933755    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz" podStartSLOduration=1.581415894 podStartE2EDuration="1.933739038s" totalImagesPullingTime="352.323144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:40.93234245 +0000 UTC m=+510.767087667" watchObservedRunningTime="2026-07-28 12:43:40.933739038 +0000 UTC m=+510.768484263"
Jul 28 12:44:11.924451 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:11.924409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc5447d6-wjknz"
Jul 28 12:44:13.045459 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.045423    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-smt2x"]
Jul 28 12:44:13.045847 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.045718    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" containerName="manager"
Jul 28 12:44:13.045847 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.045728    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" containerName="manager"
Jul 28 12:44:13.045847 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.045792    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5adee6-eb39-4dd4-89f2-1f0395c76866" containerName="manager"
Jul 28 12:44:13.049783 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.049750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.052804 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.052780    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-94qhg\""
Jul 28 12:44:13.053219 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.053194    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 12:44:13.070250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.070216    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-smt2x"]
Jul 28 12:44:13.150734 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.150689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d739936f-b0f9-4370-b3ec-cb363b1a3059-cert\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.150936 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.150791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzch7\" (UniqueName: \"kubernetes.io/projected/d739936f-b0f9-4370-b3ec-cb363b1a3059-kube-api-access-bzch7\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.251951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.251913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d739936f-b0f9-4370-b3ec-cb363b1a3059-cert\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.252134 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.251987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzch7\" (UniqueName: \"kubernetes.io/projected/d739936f-b0f9-4370-b3ec-cb363b1a3059-kube-api-access-bzch7\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.254667 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.254641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d739936f-b0f9-4370-b3ec-cb363b1a3059-cert\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.261642 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.261615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzch7\" (UniqueName: \"kubernetes.io/projected/d739936f-b0f9-4370-b3ec-cb363b1a3059-kube-api-access-bzch7\") pod \"odh-model-controller-75d9cdbddf-smt2x\" (UID: \"d739936f-b0f9-4370-b3ec-cb363b1a3059\") " pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.360787 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.360684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:13.504325 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:13.504294    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-smt2x"]
Jul 28 12:44:13.507437 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:44:13.507410    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd739936f_b0f9_4370_b3ec_cb363b1a3059.slice/crio-f33abf14e1ae073c864fb5f757ea39a8784491a8935390b5b4213b802c4193c8 WatchSource:0}: Error finding container f33abf14e1ae073c864fb5f757ea39a8784491a8935390b5b4213b802c4193c8: Status 404 returned error can't find the container with id f33abf14e1ae073c864fb5f757ea39a8784491a8935390b5b4213b802c4193c8
Jul 28 12:44:14.040826 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:14.040781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-smt2x" event={"ID":"d739936f-b0f9-4370-b3ec-cb363b1a3059","Type":"ContainerStarted","Data":"f33abf14e1ae073c864fb5f757ea39a8784491a8935390b5b4213b802c4193c8"}
Jul 28 12:44:16.057597 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:16.057561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-smt2x" event={"ID":"d739936f-b0f9-4370-b3ec-cb363b1a3059","Type":"ContainerStarted","Data":"fce4d9b518a640dec19e092c068f3e73f3aca8ea3860de203e633215bdcfb652"}
Jul 28 12:44:16.057978 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:16.057750    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:16.076493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:16.076429    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-smt2x" podStartSLOduration=1.596508367 podStartE2EDuration="4.076408264s" totalImagesPullingTime="2.479899897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:16.075211021 +0000 UTC m=+545.909956220" watchObservedRunningTime="2026-07-28 12:44:16.076408264 +0000 UTC m=+545.911153439"
Jul 28 12:44:27.065005 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.064969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-smt2x"
Jul 28 12:44:27.942292 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.942255    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-str2s"]
Jul 28 12:44:27.945748 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.945727    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-str2s"
Jul 28 12:44:27.954886 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.954848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-str2s"]
Jul 28 12:44:27.977136 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.977105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gctb\" (UniqueName: \"kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:27.977333 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:27.977143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:28.078395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.078346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gctb\" (UniqueName: \"kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:28.078395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.078398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:28.078815 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.078726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:28.091923 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.091890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gctb\" (UniqueName: \"kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb\") pod \"s3-init-str2s\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") " pod="kserve/s3-init-str2s"
Jul 28 12:44:28.271308 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.271234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-str2s"
Jul 28 12:44:28.412000 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:28.411966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-str2s"]
Jul 28 12:44:28.415909 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:44:28.415882    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55563220_8d75_4524_a5c5_4fe56a39e217.slice/crio-04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8 WatchSource:0}: Error finding container 04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8: Status 404 returned error can't find the container with id 04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8
Jul 28 12:44:29.111806 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:29.111766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerStarted","Data":"04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8"}
Jul 28 12:44:32.126316 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:32.126276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerStarted","Data":"a682653ccdc6bc4d18b95204af08f39226cb121bfeaf8a7a7e28075c7b40048b"}
Jul 28 12:44:34.135196 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:34.135161    2560 generic.go:356] "Generic (PLEG): container finished" podID="55563220-8d75-4524-a5c5-4fe56a39e217" containerID="a682653ccdc6bc4d18b95204af08f39226cb121bfeaf8a7a7e28075c7b40048b" exitCode=0
Jul 28 12:44:34.135591 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:34.135231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerDied","Data":"a682653ccdc6bc4d18b95204af08f39226cb121bfeaf8a7a7e28075c7b40048b"}
Jul 28 12:44:39.161013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:39.160979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerStarted","Data":"2a40cc83986d3e4979cb240586013af7331f2fd1a28149cc8a18ddaf1f0e676e"}
Jul 28 12:44:39.179468 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:44:39.179412    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-str2s" podStartSLOduration=2.681873856 podStartE2EDuration="12.179394566s" totalImagesPullingTime="7.41554558s" totalInitContainerRuntime="2.08197513s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:44:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:44:39.178527491 +0000 UTC m=+569.013272668" watchObservedRunningTime="2026-07-28 12:44:39.179394566 +0000 UTC m=+569.014139802"
Jul 28 12:45:10.560300 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:10.560270    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:45:10.561484 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:10.561460    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:45:44.418475 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:44.418437    2560 generic.go:356] "Generic (PLEG): container finished" podID="55563220-8d75-4524-a5c5-4fe56a39e217" containerID="2a40cc83986d3e4979cb240586013af7331f2fd1a28149cc8a18ddaf1f0e676e" exitCode=0
Jul 28 12:45:44.418939 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:44.418510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerDied","Data":"2a40cc83986d3e4979cb240586013af7331f2fd1a28149cc8a18ddaf1f0e676e"}
Jul 28 12:45:45.562564 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.562527    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-str2s"
Jul 28 12:45:45.654784 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.654742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache\") pod \"55563220-8d75-4524-a5c5-4fe56a39e217\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") "
Jul 28 12:45:45.654985 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.654800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gctb\" (UniqueName: \"kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb\") pod \"55563220-8d75-4524-a5c5-4fe56a39e217\" (UID: \"55563220-8d75-4524-a5c5-4fe56a39e217\") "
Jul 28 12:45:45.656956 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.656925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb" (OuterVolumeSpecName: "kube-api-access-7gctb") pod "55563220-8d75-4524-a5c5-4fe56a39e217" (UID: "55563220-8d75-4524-a5c5-4fe56a39e217"). InnerVolumeSpecName "kube-api-access-7gctb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:45:45.677568 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.677477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache" (OuterVolumeSpecName: "model-cache") pod "55563220-8d75-4524-a5c5-4fe56a39e217" (UID: "55563220-8d75-4524-a5c5-4fe56a39e217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:45:45.755940 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.755902    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55563220-8d75-4524-a5c5-4fe56a39e217-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:45:45.755940 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:45.755936    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gctb\" (UniqueName: \"kubernetes.io/projected/55563220-8d75-4524-a5c5-4fe56a39e217-kube-api-access-7gctb\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:45:46.428207 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:46.428165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-str2s" event={"ID":"55563220-8d75-4524-a5c5-4fe56a39e217","Type":"ContainerDied","Data":"04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8"}
Jul 28 12:45:46.428207 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:46.428207    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-str2s"
Jul 28 12:45:46.428207 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:45:46.428214    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04862a0c8371d558f9c625afbd7c786b7922efc21afba72102a8c2d0c3d1eeb8"
Jul 28 12:46:00.483745 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.483704    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:46:00.486274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.484116    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55563220-8d75-4524-a5c5-4fe56a39e217" containerName="download-hf-model"
Jul 28 12:46:00.486274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.484135    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="55563220-8d75-4524-a5c5-4fe56a39e217" containerName="download-hf-model"
Jul 28 12:46:00.486274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.484155    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55563220-8d75-4524-a5c5-4fe56a39e217" containerName="s3-init"
Jul 28 12:46:00.486274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.484166    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="55563220-8d75-4524-a5c5-4fe56a39e217" containerName="s3-init"
Jul 28 12:46:00.486274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.484299    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="55563220-8d75-4524-a5c5-4fe56a39e217" containerName="s3-init"
Jul 28 12:46:00.487404 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.487386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.490223 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.490195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:46:00.490402 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.490195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9baeb-kube-rbac-proxy-sar-config\""
Jul 28 12:46:00.490402 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.490238    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9baeb-predictor-serving-cert\""
Jul 28 12:46:00.491381 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.491359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:46:00.491510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.491450    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mdrhx\""
Jul 28 12:46:00.498210 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.498177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:46:00.578554 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.578512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2vh\" (UniqueName: \"kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.578554 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.578551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.578822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.578588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.679012 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.678975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.679187 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.679100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2vh\" (UniqueName: \"kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.679187 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.679132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.679967 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.679934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.681691 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.681667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.688972 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.688941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2vh\" (UniqueName: \"kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh\") pod \"success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") " pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.769763 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.769682    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:46:00.773508 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.773485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.776143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.776104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 28 12:46:00.776143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.776133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 28 12:46:00.776384 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.776171    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:46:00.776540 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.776507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:46:00.787146 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.787112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:46:00.798718 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.798677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:00.881022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.880689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.881022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.880737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.881022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.880781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.881022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.880817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpcb\" (UniqueName: \"kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.881022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.880892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.971244 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.971209    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:46:00.976219 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.976192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:00.979662 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.979623    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 28 12:46:00.979885 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.979634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.981931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.981988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpcb\" (UniqueName: \"kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.982067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.982125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.982168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.982174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:00.982327    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:00.982384    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls podName:2f27dff8-0f51-4b0e-9af7-7da36d17fe53 nodeName:}" failed. No retries permitted until 2026-07-28 12:46:01.482368297 +0000 UTC m=+651.317113449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls") pod "isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 12:46:00.982959 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.982723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.983774 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.983075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.983774 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.983638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:00.989245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.988381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:46:00.996392 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:00.996265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpcb\" (UniqueName: \"kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:01.073073 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.072971    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:46:01.078497 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.078465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.081118 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.081092    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 28 12:46:01.081271 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.081157    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 28 12:46:01.082933 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.082904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6c98\" (UniqueName: \"kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.083078 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.082973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.083158 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.083071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.083158 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.083122    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.083267 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.083173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.091516 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.091486    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:46:01.184237 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.184237 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4twtw\" (UniqueName: \"kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6c98\" (UniqueName: \"kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.184498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.184939 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:01.184595    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 28 12:46:01.184939 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:01.184658    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls podName:25d351f8-741a-457e-95ba-06e8863eb0ba nodeName:}" failed. No retries permitted until 2026-07-28 12:46:01.68463885 +0000 UTC m=+651.519384005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-7zt6m" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 28 12:46:01.184939 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.184704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.185311 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.185287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.185466 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.185450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.194162 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.194127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6c98\" (UniqueName: \"kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.285106 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4twtw\" (UniqueName: \"kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285349 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285628 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:01.285343    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 12:46:01.285628 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:46:01.285433    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls podName:0b1d9b25-5f04-47d9-8368-ed2d7b609be3 nodeName:}" failed. No retries permitted until 2026-07-28 12:46:01.785411461 +0000 UTC m=+651.620156617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls") pod "isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 12:46:01.285783 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.285965 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.286118 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.285982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.297292 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.297245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4twtw\" (UniqueName: \"kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.486799 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.486757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:01.489314 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.489289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-645b44c55c-znpfs\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:01.491573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.491543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerStarted","Data":"61ce5f2f28ab17b6e7325688445799fca931f5c1bf1f5ed321ae42252e1b23b6"}
Jul 28 12:46:01.687313 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.687277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:01.688708 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.688671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.691529 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.691498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-7zt6m\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.789567 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.789526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.792571 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.792534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:01.837444 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.837357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:46:01.838945 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:46:01.838909    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f27dff8_0f51_4b0e_9af7_7da36d17fe53.slice/crio-c28dc00c0c0ca29b2a57d7033114234d80276806e221ae5676bd0a5609963926 WatchSource:0}: Error finding container c28dc00c0c0ca29b2a57d7033114234d80276806e221ae5676bd0a5609963926: Status 404 returned error can't find the container with id c28dc00c0c0ca29b2a57d7033114234d80276806e221ae5676bd0a5609963926
Jul 28 12:46:01.904700 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.904452    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:01.990935 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:01.990736    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:02.140428 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.140337    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:46:02.157675 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:46:02.149403    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d351f8_741a_457e_95ba_06e8863eb0ba.slice/crio-5dd6f182fe50bf98983f3a66d08f716929663cbfa62d3a590f01c17281a38ffd WatchSource:0}: Error finding container 5dd6f182fe50bf98983f3a66d08f716929663cbfa62d3a590f01c17281a38ffd: Status 404 returned error can't find the container with id 5dd6f182fe50bf98983f3a66d08f716929663cbfa62d3a590f01c17281a38ffd
Jul 28 12:46:02.237562 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.237521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:46:02.246580 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:46:02.246538    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1d9b25_5f04_47d9_8368_ed2d7b609be3.slice/crio-f2f6107785ec5be84ba1aab2e2f576e819741f9894c3bd55477c58de2da8cdd9 WatchSource:0}: Error finding container f2f6107785ec5be84ba1aab2e2f576e819741f9894c3bd55477c58de2da8cdd9: Status 404 returned error can't find the container with id f2f6107785ec5be84ba1aab2e2f576e819741f9894c3bd55477c58de2da8cdd9
Jul 28 12:46:02.517760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.517659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerStarted","Data":"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"}
Jul 28 12:46:02.517760 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.517739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerStarted","Data":"5dd6f182fe50bf98983f3a66d08f716929663cbfa62d3a590f01c17281a38ffd"}
Jul 28 12:46:02.523909 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.523219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerStarted","Data":"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"}
Jul 28 12:46:02.523909 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.523288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerStarted","Data":"f2f6107785ec5be84ba1aab2e2f576e819741f9894c3bd55477c58de2da8cdd9"}
Jul 28 12:46:02.527922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.527611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerStarted","Data":"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"}
Jul 28 12:46:02.527922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:02.527680    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerStarted","Data":"c28dc00c0c0ca29b2a57d7033114234d80276806e221ae5676bd0a5609963926"}
Jul 28 12:46:06.558452 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:06.558403    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerID="c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b" exitCode=0
Jul 28 12:46:06.558938 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:06.558491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerDied","Data":"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"}
Jul 28 12:46:07.578470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:07.569808    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerID="27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14" exitCode=0
Jul 28 12:46:07.578470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:07.569994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerDied","Data":"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"}
Jul 28 12:46:07.578470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:07.575451    2560 generic.go:356] "Generic (PLEG): container finished" podID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerID="a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2" exitCode=0
Jul 28 12:46:07.578470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:07.575490    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerDied","Data":"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"}
Jul 28 12:46:18.644479 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:18.644389    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerStarted","Data":"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"}
Jul 28 12:46:18.649506 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:18.649421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerStarted","Data":"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"}
Jul 28 12:46:19.663772 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:19.663519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerStarted","Data":"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"}
Jul 28 12:46:23.694417 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.694377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerStarted","Data":"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"}
Jul 28 12:46:23.694928 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.694509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:23.697194 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.697152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerStarted","Data":"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"}
Jul 28 12:46:23.697366 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.697241    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:23.697366 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.697274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:23.699080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.699045    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:46:23.699498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.699450    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerStarted","Data":"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"}
Jul 28 12:46:23.699615 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.699523    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:23.719668 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.719592    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podStartSLOduration=3.112784609 podStartE2EDuration="22.719557084s" totalImagesPullingTime="15.477092874s" totalInitContainerRuntime="4.129679601s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:46:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:23.716387547 +0000 UTC m=+673.551132724" watchObservedRunningTime="2026-07-28 12:46:23.719557084 +0000 UTC m=+673.554302259"
Jul 28 12:46:23.742420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.741556    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podStartSLOduration=3.749673704 podStartE2EDuration="23.741535127s" totalImagesPullingTime="15.783811131s" totalInitContainerRuntime="4.208050292s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:46:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:23.740337902 +0000 UTC m=+673.575083109" watchObservedRunningTime="2026-07-28 12:46:23.741535127 +0000 UTC m=+673.576280302"
Jul 28 12:46:23.766514 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:23.766447    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podStartSLOduration=1.699639176 podStartE2EDuration="23.766427593s" totalImagesPullingTime="22.066788417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:46:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:23.763836902 +0000 UTC m=+673.598582075" watchObservedRunningTime="2026-07-28 12:46:23.766427593 +0000 UTC m=+673.601172766"
Jul 28 12:46:24.706573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:24.706475    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:46:24.706573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:24.706505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:24.706573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:24.706581    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:24.707250 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:24.707224    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:46:24.707334 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:24.707294    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:46:25.711497 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:25.711444    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:46:25.711993 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:25.711600    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:46:29.712183 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:29.712138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:46:29.712805 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:29.712769    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:46:30.716143 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:30.716113    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:46:30.716632 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:30.716478    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:46:30.716753 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:30.716723    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:46:30.717037 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:30.717004    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:46:37.787305 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:37.787265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerStarted","Data":"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"}
Jul 28 12:46:37.787837 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:37.787338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerStarted","Data":"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"}
Jul 28 12:46:37.787837 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:37.787553    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:37.809510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:37.809456    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podStartSLOduration=4.281483089 podStartE2EDuration="37.80943841s" totalImagesPullingTime="29.450587144s" totalInitContainerRuntime="4.077368177s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:37.807688783 +0000 UTC m=+687.642433998" watchObservedRunningTime="2026-07-28 12:46:37.80943841 +0000 UTC m=+687.644183584"
Jul 28 12:46:38.791549 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:38.791517    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:38.792922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:38.792894    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:39.713614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:39.713577    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:46:39.796472 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:39.796429    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:40.716737 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:40.716690    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:46:40.717026 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:40.716984    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:46:44.801792 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:44.801758    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:46:44.802312 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:44.802285    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:49.713660 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:49.713617    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:46:50.717174 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:50.717131    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:46:50.717619 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:50.717132    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:46:54.802889 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:54.802839    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:59.712889 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:46:59.712821    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:47:00.716732 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:00.716693    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 12:47:00.717149 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:00.717004    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:47:04.803013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:04.802914    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:09.712636 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:09.712595    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:47:10.717464 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:10.717411    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:47:10.718021 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:10.718000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:47:14.802586 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:14.802548    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:19.713369 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:19.713321    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:47:20.717510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:20.717465    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:47:24.803080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:24.803035    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:29.712840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:29.712794    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:47:30.717035 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:30.716996    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:47:34.803235 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:34.803189    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:39.712933 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:39.712895    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:47:40.717420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:40.717372    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 12:47:41.111679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.111582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:47:41.114774 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.114741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.118209 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.118165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc405-predictor-serving-cert\""
Jul 28 12:47:41.118496 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.118471    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc405-kube-rbac-proxy-sar-config\""
Jul 28 12:47:41.135813 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.135775    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:47:41.168724 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.168687    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:47:41.169044 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.169005    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container" containerID="cri-o://cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801" gracePeriod=30
Jul 28 12:47:41.169170 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.169059    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kube-rbac-proxy" containerID="cri-o://fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e" gracePeriod=30
Jul 28 12:47:41.227198 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.227161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljjd\" (UniqueName: \"kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.227395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.227225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.227395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.227312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.328560 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.328521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.328770 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.328622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljjd\" (UniqueName: \"kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.328770 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.328669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.328900 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:47:41.328787    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fc405-predictor-serving-cert: secret "success-200-isvc-fc405-predictor-serving-cert" not found
Jul 28 12:47:41.328900 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:47:41.328892    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls podName:5ed5813f-1518-42fa-aeaf-a6352bc37feb nodeName:}" failed. No retries permitted until 2026-07-28 12:47:41.828845798 +0000 UTC m=+751.663590951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls") pod "success-200-isvc-fc405-predictor-664b4c95f9-mptpw" (UID: "5ed5813f-1518-42fa-aeaf-a6352bc37feb") : secret "success-200-isvc-fc405-predictor-serving-cert" not found
Jul 28 12:47:41.329426 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.329404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.342163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.342127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljjd\" (UniqueName: \"kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.833509 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.833463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:41.835934 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:41.835911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") pod \"success-200-isvc-fc405-predictor-664b4c95f9-mptpw\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") " pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:42.028236 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:42.028191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:42.090273 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:42.090124    2560 generic.go:356] "Generic (PLEG): container finished" podID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerID="fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e" exitCode=2
Jul 28 12:47:42.090273 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:42.090170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerDied","Data":"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"}
Jul 28 12:47:42.174017 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:42.173981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:47:42.180424 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:47:42.180394    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed5813f_1518_42fa_aeaf_a6352bc37feb.slice/crio-8f00159bd8b22d6bd435865a458e8f63c4c28877ec8bf876238bf4ca69355ff0 WatchSource:0}: Error finding container 8f00159bd8b22d6bd435865a458e8f63c4c28877ec8bf876238bf4ca69355ff0: Status 404 returned error can't find the container with id 8f00159bd8b22d6bd435865a458e8f63c4c28877ec8bf876238bf4ca69355ff0
Jul 28 12:47:42.661471 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:42.661434    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:47:43.095920 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.095854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerStarted","Data":"a1aeed57b349061e67a7be22710dbe0680caaf38471002385bdefb3caf50468c"}
Jul 28 12:47:43.096420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.095950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerStarted","Data":"c67779918fb4c77854f48286b9620b6e65baaf85f55b8875a3b8c13abfd7c674"}
Jul 28 12:47:43.096420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.095986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:43.096420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.096010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerStarted","Data":"8f00159bd8b22d6bd435865a458e8f63c4c28877ec8bf876238bf4ca69355ff0"}
Jul 28 12:47:43.096420 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.096258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:43.097504 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.097478    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:47:43.116976 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:43.116926    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podStartSLOduration=2.116908207 podStartE2EDuration="2.116908207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:47:43.115399298 +0000 UTC m=+752.950144497" watchObservedRunningTime="2026-07-28 12:47:43.116908207 +0000 UTC m=+752.951653381"
Jul 28 12:47:44.100554 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.100509    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:47:44.525723 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.525694    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:47:44.659215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.659121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config\") pod \"fcce6d64-f567-4c44-ade5-098cf14ea233\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") "
Jul 28 12:47:44.659388 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.659226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc2vh\" (UniqueName: \"kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh\") pod \"fcce6d64-f567-4c44-ade5-098cf14ea233\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") "
Jul 28 12:47:44.659388 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.659267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls\") pod \"fcce6d64-f567-4c44-ade5-098cf14ea233\" (UID: \"fcce6d64-f567-4c44-ade5-098cf14ea233\") "
Jul 28 12:47:44.659535 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.659505    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcce6d64-f567-4c44-ade5-098cf14ea233" (UID: "fcce6d64-f567-4c44-ade5-098cf14ea233"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:47:44.661665 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.661635    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh" (OuterVolumeSpecName: "kube-api-access-xc2vh") pod "fcce6d64-f567-4c44-ade5-098cf14ea233" (UID: "fcce6d64-f567-4c44-ade5-098cf14ea233"). InnerVolumeSpecName "kube-api-access-xc2vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:47:44.661665 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.661634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcce6d64-f567-4c44-ade5-098cf14ea233" (UID: "fcce6d64-f567-4c44-ade5-098cf14ea233"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:47:44.760591 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.760553    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcce6d64-f567-4c44-ade5-098cf14ea233-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:47:44.760591 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.760593    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:47:44.760830 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.760608    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc2vh\" (UniqueName: \"kubernetes.io/projected/fcce6d64-f567-4c44-ade5-098cf14ea233-kube-api-access-xc2vh\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:47:44.803738 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:44.803707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:47:45.105704 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.105669    2560 generic.go:356] "Generic (PLEG): container finished" podID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerID="cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801" exitCode=0
Jul 28 12:47:45.106153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.105722    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"
Jul 28 12:47:45.106153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.105752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerDied","Data":"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"}
Jul 28 12:47:45.106153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.105806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j" event={"ID":"fcce6d64-f567-4c44-ade5-098cf14ea233","Type":"ContainerDied","Data":"61ce5f2f28ab17b6e7325688445799fca931f5c1bf1f5ed321ae42252e1b23b6"}
Jul 28 12:47:45.106153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.105834    2560 scope.go:117] "RemoveContainer" containerID="fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"
Jul 28 12:47:45.106370 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.106274    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:47:45.114581 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.114560    2560 scope.go:117] "RemoveContainer" containerID="cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"
Jul 28 12:47:45.122800 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.122728    2560 scope.go:117] "RemoveContainer" containerID="fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"
Jul 28 12:47:45.123245 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:47:45.123132    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e\": container with ID starting with fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e not found: ID does not exist" containerID="fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"
Jul 28 12:47:45.123245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.123198    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e"} err="failed to get container status \"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e\": rpc error: code = NotFound desc = could not find container \"fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e\": container with ID starting with fcd2a08ac94516b2487c08ddc86ef33fc31c25ee34983b3b861239df7cd67f8e not found: ID does not exist"
Jul 28 12:47:45.123245 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.123222    2560 scope.go:117] "RemoveContainer" containerID="cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"
Jul 28 12:47:45.123565 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:47:45.123546    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801\": container with ID starting with cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801 not found: ID does not exist" containerID="cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"
Jul 28 12:47:45.123619 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.123583    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801"} err="failed to get container status \"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801\": rpc error: code = NotFound desc = could not find container \"cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801\": container with ID starting with cc985ee18a92eb45334844af5a73d6f3095889f4360802d6fef19de9142ec801 not found: ID does not exist"
Jul 28 12:47:45.124905 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.124884    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:47:45.128464 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:45.128441    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9baeb-predictor-6f9fb747fc-4td2j"]
Jul 28 12:47:46.661725 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:46.661683    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" path="/var/lib/kubelet/pods/fcce6d64-f567-4c44-ade5-098cf14ea233/volumes"
Jul 28 12:47:48.661520 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:48.661494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:47:50.111401 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:50.111369    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:47:50.111849 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:47:50.111824    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:48:00.112764 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:00.112721    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:48:10.111819 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:10.111778    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:48:20.112047 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.112005    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 12:48:20.766532 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.766495    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:48:20.766833 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.766809    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" containerID="cri-o://425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f" gracePeriod=30
Jul 28 12:48:20.766900 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.766825    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kube-rbac-proxy" containerID="cri-o://1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8" gracePeriod=30
Jul 28 12:48:20.822136 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.821993    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:48:20.822548 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.822508    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container" containerID="cri-o://e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce" gracePeriod=30
Jul 28 12:48:20.822679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.822513    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kube-rbac-proxy" containerID="cri-o://ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11" gracePeriod=30
Jul 28 12:48:20.883507 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.883473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:48:20.883769 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.883732    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" containerID="cri-o://36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758" gracePeriod=30
Jul 28 12:48:20.884040 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.883780    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kube-rbac-proxy" containerID="cri-o://03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923" gracePeriod=30
Jul 28 12:48:20.887539 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.887498    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:48:20.888090 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888066    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kube-rbac-proxy"
Jul 28 12:48:20.888193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kube-rbac-proxy"
Jul 28 12:48:20.888193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888130    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container"
Jul 28 12:48:20.888193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888139    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container"
Jul 28 12:48:20.888385 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888257    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kube-rbac-proxy"
Jul 28 12:48:20.888385 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.888275    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcce6d64-f567-4c44-ade5-098cf14ea233" containerName="kserve-container"
Jul 28 12:48:20.891553 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.891534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:20.894112 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.894091    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7fa9-predictor-serving-cert\""
Jul 28 12:48:20.894222 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.894157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7fa9-kube-rbac-proxy-sar-config\""
Jul 28 12:48:20.904524 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.904262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:48:20.980090 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.980053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:20.980287 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.980103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:20.980287 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:20.980150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms6q\" (UniqueName: \"kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.081531 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.081420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.081531 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.081470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.081531 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.081490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms6q\" (UniqueName: \"kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.081813 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:21.081657    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-serving-cert: secret "success-200-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:21.081813 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:21.081760    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls podName:7bc0fda4-9a4b-42c3-adb8-d78a4d95615c nodeName:}" failed. No retries permitted until 2026-07-28 12:48:21.581739006 +0000 UTC m=+791.416484162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls") pod "success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" (UID: "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c") : secret "success-200-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:21.082265 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.082241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.090700 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.090675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms6q\" (UniqueName: \"kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.283466 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.283428    2560 generic.go:356] "Generic (PLEG): container finished" podID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerID="1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8" exitCode=2
Jul 28 12:48:21.283951 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.283508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerDied","Data":"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"}
Jul 28 12:48:21.285899 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.285876    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerID="ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11" exitCode=2
Jul 28 12:48:21.286022 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.285936    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerDied","Data":"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"}
Jul 28 12:48:21.288097 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.288074    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerID="03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923" exitCode=2
Jul 28 12:48:21.288191 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.288135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerDied","Data":"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"}
Jul 28 12:48:21.585371 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:21.585329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:21.585557 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:21.585472    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-serving-cert: secret "success-200-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:21.585557 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:21.585535    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls podName:7bc0fda4-9a4b-42c3-adb8-d78a4d95615c nodeName:}" failed. No retries permitted until 2026-07-28 12:48:22.585516925 +0000 UTC m=+792.420262085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls") pod "success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" (UID: "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c") : secret "success-200-isvc-f7fa9-predictor-serving-cert" not found
Jul 28 12:48:22.596199 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:22.596159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:22.598572 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:22.598539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") " pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:22.656632 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:22.656579    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 12:48:22.712638 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:22.712598    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:22.867597 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:22.867421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:48:22.870524 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:48:22.870497    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc0fda4_9a4b_42c3_adb8_d78a4d95615c.slice/crio-0ffea86a20eace6c347d4c91c399af968c48f1f38ba5c42ac950440d5158fd7c WatchSource:0}: Error finding container 0ffea86a20eace6c347d4c91c399af968c48f1f38ba5c42ac950440d5158fd7c: Status 404 returned error can't find the container with id 0ffea86a20eace6c347d4c91c399af968c48f1f38ba5c42ac950440d5158fd7c
Jul 28 12:48:23.300078 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.300032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerStarted","Data":"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"}
Jul 28 12:48:23.300284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.300091    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerStarted","Data":"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"}
Jul 28 12:48:23.300284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.300119    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:23.300284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.300137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerStarted","Data":"0ffea86a20eace6c347d4c91c399af968c48f1f38ba5c42ac950440d5158fd7c"}
Jul 28 12:48:23.300284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.300175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:23.301590 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.301555    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:48:23.320742 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:23.320684    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podStartSLOduration=3.320666343 podStartE2EDuration="3.320666343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:23.320258156 +0000 UTC m=+793.155003331" watchObservedRunningTime="2026-07-28 12:48:23.320666343 +0000 UTC m=+793.155411563"
Jul 28 12:48:24.309612 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:24.309428    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:48:24.707117 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:24.707077    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 12:48:24.797568 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:24.797526    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:48:24.803138 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:24.803103    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:48:24.915060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:24.915035    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:48:25.016899 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.016783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") pod \"25d351f8-741a-457e-95ba-06e8863eb0ba\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") "
Jul 28 12:48:25.016899 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.016832    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location\") pod \"25d351f8-741a-457e-95ba-06e8863eb0ba\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") "
Jul 28 12:48:25.016899 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.016849    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert\") pod \"25d351f8-741a-457e-95ba-06e8863eb0ba\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") "
Jul 28 12:48:25.017202 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.016910    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config\") pod \"25d351f8-741a-457e-95ba-06e8863eb0ba\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") "
Jul 28 12:48:25.017202 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.016936    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6c98\" (UniqueName: \"kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98\") pod \"25d351f8-741a-457e-95ba-06e8863eb0ba\" (UID: \"25d351f8-741a-457e-95ba-06e8863eb0ba\") "
Jul 28 12:48:25.017202 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.017110    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25d351f8-741a-457e-95ba-06e8863eb0ba" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:48:25.017341 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.017230    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d351f8-741a-457e-95ba-06e8863eb0ba-kserve-provision-location\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:25.017341 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.017244    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25d351f8-741a-457e-95ba-06e8863eb0ba" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:25.017341 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.017273    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25d351f8-741a-457e-95ba-06e8863eb0ba" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:25.019013 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.018985    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25d351f8-741a-457e-95ba-06e8863eb0ba" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:25.019124 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.019085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98" (OuterVolumeSpecName: "kube-api-access-c6c98") pod "25d351f8-741a-457e-95ba-06e8863eb0ba" (UID: "25d351f8-741a-457e-95ba-06e8863eb0ba"). InnerVolumeSpecName "kube-api-access-c6c98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:48:25.117765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.117734    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d351f8-741a-457e-95ba-06e8863eb0ba-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:25.117765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.117761    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-cabundle-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:25.117765 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.117770    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:25.118005 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.117780    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6c98\" (UniqueName: \"kubernetes.io/projected/25d351f8-741a-457e-95ba-06e8863eb0ba-kube-api-access-c6c98\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:25.311922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.311813    2560 generic.go:356] "Generic (PLEG): container finished" podID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerID="425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f" exitCode=0
Jul 28 12:48:25.311922 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.311895    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"
Jul 28 12:48:25.312360 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.311895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerDied","Data":"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"}
Jul 28 12:48:25.312360 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.312019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m" event={"ID":"25d351f8-741a-457e-95ba-06e8863eb0ba","Type":"ContainerDied","Data":"5dd6f182fe50bf98983f3a66d08f716929663cbfa62d3a590f01c17281a38ffd"}
Jul 28 12:48:25.312360 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.312059    2560 scope.go:117] "RemoveContainer" containerID="1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"
Jul 28 12:48:25.320390 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.320368    2560 scope.go:117] "RemoveContainer" containerID="425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"
Jul 28 12:48:25.328025 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.328007    2560 scope.go:117] "RemoveContainer" containerID="a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"
Jul 28 12:48:25.335101 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335080    2560 scope.go:117] "RemoveContainer" containerID="1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"
Jul 28 12:48:25.335350 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:25.335327    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8\": container with ID starting with 1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8 not found: ID does not exist" containerID="1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"
Jul 28 12:48:25.335396 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335359    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8"} err="failed to get container status \"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8\": rpc error: code = NotFound desc = could not find container \"1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8\": container with ID starting with 1f980fc4f23edbcc5613d14129553c146770e2583678ee373819630bd78081f8 not found: ID does not exist"
Jul 28 12:48:25.335396 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335377    2560 scope.go:117] "RemoveContainer" containerID="425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"
Jul 28 12:48:25.335616 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:25.335594    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f\": container with ID starting with 425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f not found: ID does not exist" containerID="425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"
Jul 28 12:48:25.335679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335626    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f"} err="failed to get container status \"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f\": rpc error: code = NotFound desc = could not find container \"425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f\": container with ID starting with 425200c155533e5dc2a34a9dee240413d7af230559d574c82792f435d5818b5f not found: ID does not exist"
Jul 28 12:48:25.335679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335650    2560 scope.go:117] "RemoveContainer" containerID="a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"
Jul 28 12:48:25.335899 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:25.335880    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2\": container with ID starting with a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2 not found: ID does not exist" containerID="a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"
Jul 28 12:48:25.335935 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.335903    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2"} err="failed to get container status \"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2\": rpc error: code = NotFound desc = could not find container \"a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2\": container with ID starting with a4f29bc5e33c87b90659517c00bbf8aa720f18210c41071978a469d982430cb2 not found: ID does not exist"
Jul 28 12:48:25.341316 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.341291    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:48:25.346990 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.346962    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-7zt6m"]
Jul 28 12:48:25.712386 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:25.712350    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:48:26.662884 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.662621    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" path="/var/lib/kubelet/pods/25d351f8-741a-457e-95ba-06e8863eb0ba/volumes"
Jul 28 12:48:26.798952 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.798887    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:48:26.800936 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.800918    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:48:26.934293 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfpcb\" (UniqueName: \"kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb\") pod \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") "
Jul 28 12:48:26.934493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert\") pod \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") "
Jul 28 12:48:26.934493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") pod \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") "
Jul 28 12:48:26.934493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934370    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config\") pod \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") "
Jul 28 12:48:26.934493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert\") pod \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") "
Jul 28 12:48:26.934493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4twtw\" (UniqueName: \"kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw\") pod \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") "
Jul 28 12:48:26.934766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location\") pod \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") "
Jul 28 12:48:26.934766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") pod \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") "
Jul 28 12:48:26.934766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934584    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location\") pod \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\" (UID: \"2f27dff8-0f51-4b0e-9af7-7da36d17fe53\") "
Jul 28 12:48:26.934766 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.934609    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config\") pod \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\" (UID: \"0b1d9b25-5f04-47d9-8368-ed2d7b609be3\") "
Jul 28 12:48:26.935270 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f27dff8-0f51-4b0e-9af7-7da36d17fe53" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:26.935270 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b1d9b25-5f04-47d9-8368-ed2d7b609be3" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:26.935270 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935022    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b1d9b25-5f04-47d9-8368-ed2d7b609be3" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:48:26.935270 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935174    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f27dff8-0f51-4b0e-9af7-7da36d17fe53" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:26.935270 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935215    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b1d9b25-5f04-47d9-8368-ed2d7b609be3" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:48:26.935535 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.935273    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f27dff8-0f51-4b0e-9af7-7da36d17fe53" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:48:26.936710 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.936686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f27dff8-0f51-4b0e-9af7-7da36d17fe53" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:26.936930 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.936907    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb" (OuterVolumeSpecName: "kube-api-access-vfpcb") pod "2f27dff8-0f51-4b0e-9af7-7da36d17fe53" (UID: "2f27dff8-0f51-4b0e-9af7-7da36d17fe53"). InnerVolumeSpecName "kube-api-access-vfpcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:48:26.937051 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.936990    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b1d9b25-5f04-47d9-8368-ed2d7b609be3" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:48:26.937110 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:26.937048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw" (OuterVolumeSpecName: "kube-api-access-4twtw") pod "0b1d9b25-5f04-47d9-8368-ed2d7b609be3" (UID: "0b1d9b25-5f04-47d9-8368-ed2d7b609be3"). InnerVolumeSpecName "kube-api-access-4twtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:48:27.036174 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036144    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036174 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036174    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kserve-provision-location\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036188    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036202    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfpcb\" (UniqueName: \"kubernetes.io/projected/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-api-access-vfpcb\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036214    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-cabundle-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036231    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036242    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f27dff8-0f51-4b0e-9af7-7da36d17fe53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036254    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-cabundle-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036265    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4twtw\" (UniqueName: \"kubernetes.io/projected/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kube-api-access-4twtw\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.036403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.036276    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1d9b25-5f04-47d9-8368-ed2d7b609be3-kserve-provision-location\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:48:27.324716 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.324681    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerID="e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce" exitCode=0
Jul 28 12:48:27.324948 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.324733    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"
Jul 28 12:48:27.324948 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.324761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerDied","Data":"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"}
Jul 28 12:48:27.324948 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.324829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh" event={"ID":"0b1d9b25-5f04-47d9-8368-ed2d7b609be3","Type":"ContainerDied","Data":"f2f6107785ec5be84ba1aab2e2f576e819741f9894c3bd55477c58de2da8cdd9"}
Jul 28 12:48:27.324948 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.324890    2560 scope.go:117] "RemoveContainer" containerID="ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"
Jul 28 12:48:27.326847 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.326823    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerID="36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758" exitCode=0
Jul 28 12:48:27.326967 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.326893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerDied","Data":"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"}
Jul 28 12:48:27.326967 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.326909    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"
Jul 28 12:48:27.326967 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.326955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs" event={"ID":"2f27dff8-0f51-4b0e-9af7-7da36d17fe53","Type":"ContainerDied","Data":"c28dc00c0c0ca29b2a57d7033114234d80276806e221ae5676bd0a5609963926"}
Jul 28 12:48:27.336454 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.336434    2560 scope.go:117] "RemoveContainer" containerID="e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"
Jul 28 12:48:27.347529 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.347508    2560 scope.go:117] "RemoveContainer" containerID="27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"
Jul 28 12:48:27.355153 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.355118    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:48:27.356598 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.356580    2560 scope.go:117] "RemoveContainer" containerID="ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"
Jul 28 12:48:27.356933 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.356859    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11\": container with ID starting with ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11 not found: ID does not exist" containerID="ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"
Jul 28 12:48:27.357009 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.356947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11"} err="failed to get container status \"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11\": rpc error: code = NotFound desc = could not find container \"ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11\": container with ID starting with ebe14dac42b73189afdab96211fd987554e99b052d275ac735f60db17c001a11 not found: ID does not exist"
Jul 28 12:48:27.357009 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.356974    2560 scope.go:117] "RemoveContainer" containerID="e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"
Jul 28 12:48:27.357233 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.357215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce\": container with ID starting with e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce not found: ID does not exist" containerID="e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"
Jul 28 12:48:27.357276 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.357240    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce"} err="failed to get container status \"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce\": rpc error: code = NotFound desc = could not find container \"e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce\": container with ID starting with e4bba9806c7697a6d27947a9fb030b8e95f2975b6f46240cc1475c03a5b8a2ce not found: ID does not exist"
Jul 28 12:48:27.357276 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.357256    2560 scope.go:117] "RemoveContainer" containerID="27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"
Jul 28 12:48:27.357478 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.357463    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14\": container with ID starting with 27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14 not found: ID does not exist" containerID="27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"
Jul 28 12:48:27.357519 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.357483    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14"} err="failed to get container status \"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14\": rpc error: code = NotFound desc = could not find container \"27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14\": container with ID starting with 27046b2983d573635a2eff578689b288d214d483a4791055cf58b05325b6cb14 not found: ID does not exist"
Jul 28 12:48:27.357519 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.357495    2560 scope.go:117] "RemoveContainer" containerID="03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"
Jul 28 12:48:27.363451 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.363430    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-544798b8f5-kq6xh"]
Jul 28 12:48:27.365282 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.365267    2560 scope.go:117] "RemoveContainer" containerID="36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"
Jul 28 12:48:27.372267 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.372250    2560 scope.go:117] "RemoveContainer" containerID="c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"
Jul 28 12:48:27.377147 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.377128    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:48:27.379981 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.379960    2560 scope.go:117] "RemoveContainer" containerID="03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"
Jul 28 12:48:27.380269 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.380249    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923\": container with ID starting with 03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923 not found: ID does not exist" containerID="03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"
Jul 28 12:48:27.380324 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.380276    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923"} err="failed to get container status \"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923\": rpc error: code = NotFound desc = could not find container \"03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923\": container with ID starting with 03cf0cb5e4b584d327530e1f0484ac39ea26d4930f834ce16140d79f46001923 not found: ID does not exist"
Jul 28 12:48:27.380324 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.380294    2560 scope.go:117] "RemoveContainer" containerID="36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"
Jul 28 12:48:27.380545 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.380524    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758\": container with ID starting with 36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758 not found: ID does not exist" containerID="36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"
Jul 28 12:48:27.380599 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.380552    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758"} err="failed to get container status \"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758\": rpc error: code = NotFound desc = could not find container \"36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758\": container with ID starting with 36b33d50fa67112bf7f2ca7c8203cb05ef20630b77591b0ef81b1fe90d450758 not found: ID does not exist"
Jul 28 12:48:27.380599 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.380569    2560 scope.go:117] "RemoveContainer" containerID="c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"
Jul 28 12:48:27.380794 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:48:27.380776    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b\": container with ID starting with c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b not found: ID does not exist" containerID="c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"
Jul 28 12:48:27.380834 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.380799    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b"} err="failed to get container status \"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b\": rpc error: code = NotFound desc = could not find container \"c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b\": container with ID starting with c71c9469c60eb99d0b97ef8a331d0adff540e759cda220024b78604a8fdda22b not found: ID does not exist"
Jul 28 12:48:27.383524 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:27.383503    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-645b44c55c-znpfs"]
Jul 28 12:48:28.667295 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:28.667262    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" path="/var/lib/kubelet/pods/0b1d9b25-5f04-47d9-8368-ed2d7b609be3/volumes"
Jul 28 12:48:28.667745 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:28.667731    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" path="/var/lib/kubelet/pods/2f27dff8-0f51-4b0e-9af7-7da36d17fe53/volumes"
Jul 28 12:48:29.311821 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:29.311792    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:48:29.312403 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:29.312369    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:48:30.112794 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:30.112756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:48:39.312745 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:39.312658    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:48:49.313115 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:49.313072    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:48:59.313237 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:48:59.313194    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:09.312588 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:49:09.312544    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:19.313642 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:49:19.313608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:50:10.586822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:50:10.586747    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:50:10.586822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:50:10.586768    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:55:10.612939 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:55:10.612838    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:55:10.619397 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:55:10.614074    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 12:56:55.924111 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.924070    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:56:55.924751 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.924319    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container" containerID="cri-o://c67779918fb4c77854f48286b9620b6e65baaf85f55b8875a3b8c13abfd7c674" gracePeriod=30
Jul 28 12:56:55.924751 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.924351    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kube-rbac-proxy" containerID="cri-o://a1aeed57b349061e67a7be22710dbe0680caaf38471002385bdefb3caf50468c" gracePeriod=30
Jul 28 12:56:55.983667 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.983624    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:56:55.984120 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984096    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container"
Jul 28 12:56:55.984120 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984143    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984152    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984160    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984168    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984205    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984224    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984238    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984246    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984260    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984268    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="storage-initializer"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984281    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984289    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984298    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container"
Jul 28 12:56:55.984303 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984306    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984383    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kserve-container"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984391    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984398    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1d9b25-5f04-47d9-8368-ed2d7b609be3" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984405    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kserve-container"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984411    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f27dff8-0f51-4b0e-9af7-7da36d17fe53" containerName="kube-rbac-proxy"
Jul 28 12:56:55.984995 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.984417    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d351f8-741a-457e-95ba-06e8863eb0ba" containerName="kserve-container"
Jul 28 12:56:55.989786 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.989754    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:55.993242 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.993215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-75e21-predictor-serving-cert\""
Jul 28 12:56:55.993757 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:55.993524    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-75e21-kube-rbac-proxy-sar-config\""
Jul 28 12:56:56.014355 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.014307    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:56:56.112662 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.112621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.112910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.112704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.112910 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.112735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xj2\" (UniqueName: \"kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.214179 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.214127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.214402 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.214196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xj2\" (UniqueName: \"kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.214402 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.214238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.214402 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:56:56.214298    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-75e21-predictor-serving-cert: secret "success-200-isvc-75e21-predictor-serving-cert" not found
Jul 28 12:56:56.214402 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:56:56.214379    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls podName:6c748ae6-3fed-4d4e-8cc0-57fa282a85d9 nodeName:}" failed. No retries permitted until 2026-07-28 12:56:56.714357289 +0000 UTC m=+1306.549102446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls") pod "success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" (UID: "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9") : secret "success-200-isvc-75e21-predictor-serving-cert" not found
Jul 28 12:56:56.214972 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.214950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.223892 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.223821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xj2\" (UniqueName: \"kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.416193 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.416162    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerID="a1aeed57b349061e67a7be22710dbe0680caaf38471002385bdefb3caf50468c" exitCode=2
Jul 28 12:56:56.416367 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.416206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerDied","Data":"a1aeed57b349061e67a7be22710dbe0680caaf38471002385bdefb3caf50468c"}
Jul 28 12:56:56.718346 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.718299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.720783 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.720745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") pod \"success-200-isvc-75e21-predictor-7df67ccd87-mnfc8\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") " pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:56.913425 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:56.913387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:57.057001 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.056973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:56:57.059295 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:56:57.059262    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c748ae6_3fed_4d4e_8cc0_57fa282a85d9.slice/crio-6e7d72b2dff030c6ec123cf450a84f4aa0efc94afc6e4a89392c8406ceaa1fcd WatchSource:0}: Error finding container 6e7d72b2dff030c6ec123cf450a84f4aa0efc94afc6e4a89392c8406ceaa1fcd: Status 404 returned error can't find the container with id 6e7d72b2dff030c6ec123cf450a84f4aa0efc94afc6e4a89392c8406ceaa1fcd
Jul 28 12:56:57.422122 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.422026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerStarted","Data":"8b28bcd10e3c6f848ce46f9f4473a35bb99a4eac4429c31b9918e03e4e9e7237"}
Jul 28 12:56:57.422122 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.422090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerStarted","Data":"38763508d405b2eb01ea5a604bb034b7f112d02ef3960c7b6a9cab8f3dd4899e"}
Jul 28 12:56:57.422122 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.422113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerStarted","Data":"6e7d72b2dff030c6ec123cf450a84f4aa0efc94afc6e4a89392c8406ceaa1fcd"}
Jul 28 12:56:57.422413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.422143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:57.422413 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.422168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:56:57.423538 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.423511    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:56:57.442234 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:57.442163    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podStartSLOduration=2.442138294 podStartE2EDuration="2.442138294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:57.440503475 +0000 UTC m=+1307.275248651" watchObservedRunningTime="2026-07-28 12:56:57.442138294 +0000 UTC m=+1307.276883468"
Jul 28 12:56:58.428788 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:58.428741    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:56:59.434913 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.434844    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerID="c67779918fb4c77854f48286b9620b6e65baaf85f55b8875a3b8c13abfd7c674" exitCode=0
Jul 28 12:56:59.434913 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.434899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerDied","Data":"c67779918fb4c77854f48286b9620b6e65baaf85f55b8875a3b8c13abfd7c674"}
Jul 28 12:56:59.485947 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.485920    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:56:59.645281 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.645181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jljjd\" (UniqueName: \"kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd\") pod \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") "
Jul 28 12:56:59.645281 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.645243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config\") pod \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") "
Jul 28 12:56:59.645510 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.645351    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") pod \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\" (UID: \"5ed5813f-1518-42fa-aeaf-a6352bc37feb\") "
Jul 28 12:56:59.645567 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.645540    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ed5813f-1518-42fa-aeaf-a6352bc37feb" (UID: "5ed5813f-1518-42fa-aeaf-a6352bc37feb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:59.647443 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.647416    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ed5813f-1518-42fa-aeaf-a6352bc37feb" (UID: "5ed5813f-1518-42fa-aeaf-a6352bc37feb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:59.647527 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.647433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd" (OuterVolumeSpecName: "kube-api-access-jljjd") pod "5ed5813f-1518-42fa-aeaf-a6352bc37feb" (UID: "5ed5813f-1518-42fa-aeaf-a6352bc37feb"). InnerVolumeSpecName "kube-api-access-jljjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:59.746215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.746170    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ed5813f-1518-42fa-aeaf-a6352bc37feb-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:56:59.746215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.746204    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljjd\" (UniqueName: \"kubernetes.io/projected/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-api-access-jljjd\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:56:59.746215 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:56:59.746215    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ed5813f-1518-42fa-aeaf-a6352bc37feb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:57:00.442073 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.442043    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"
Jul 28 12:57:00.442499 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.442072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw" event={"ID":"5ed5813f-1518-42fa-aeaf-a6352bc37feb","Type":"ContainerDied","Data":"8f00159bd8b22d6bd435865a458e8f63c4c28877ec8bf876238bf4ca69355ff0"}
Jul 28 12:57:00.442499 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.442139    2560 scope.go:117] "RemoveContainer" containerID="a1aeed57b349061e67a7be22710dbe0680caaf38471002385bdefb3caf50468c"
Jul 28 12:57:00.451529 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.451504    2560 scope.go:117] "RemoveContainer" containerID="c67779918fb4c77854f48286b9620b6e65baaf85f55b8875a3b8c13abfd7c674"
Jul 28 12:57:00.469274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.469241    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:57:00.475813 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.475783    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc405-predictor-664b4c95f9-mptpw"]
Jul 28 12:57:00.662340 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:00.662306    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" path="/var/lib/kubelet/pods/5ed5813f-1518-42fa-aeaf-a6352bc37feb/volumes"
Jul 28 12:57:03.434434 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:03.434043    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:57:03.435041 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:03.434574    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:57:13.434890 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:13.434822    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:57:23.435545 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:23.435495    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:57:33.435170 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:33.435074    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:57:43.436020 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:43.435988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:57:45.699080 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.699042    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:57:45.699526 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.699272    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container" containerID="cri-o://b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72" gracePeriod=30
Jul 28 12:57:45.699526 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.699336    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kube-rbac-proxy" containerID="cri-o://180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd" gracePeriod=30
Jul 28 12:57:45.772592 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:57:45.772904 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container"
Jul 28 12:57:45.772976 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772906    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container"
Jul 28 12:57:45.772976 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772926    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kube-rbac-proxy"
Jul 28 12:57:45.772976 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772931    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kube-rbac-proxy"
Jul 28 12:57:45.773126 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.772994    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kube-rbac-proxy"
Jul 28 12:57:45.773126 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.773005    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed5813f-1518-42fa-aeaf-a6352bc37feb" containerName="kserve-container"
Jul 28 12:57:45.776654 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.776630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:45.779473 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.779444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc1de-predictor-serving-cert\""
Jul 28 12:57:45.779473 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.779461    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc1de-kube-rbac-proxy-sar-config\""
Jul 28 12:57:45.787001 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.786975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:57:45.923712 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.923667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:45.923903 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.923726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjtt\" (UniqueName: \"kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:45.923903 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:45.923820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.025075 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.024976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.025075 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.025064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.025316 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.025107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjtt\" (UniqueName: \"kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.025316 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:57:46.025153    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-serving-cert: secret "success-200-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:46.025316 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:57:46.025229    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls podName:78b47d4c-aba1-41e4-ab30-fed27babeb3b nodeName:}" failed. No retries permitted until 2026-07-28 12:57:46.525207877 +0000 UTC m=+1356.359953042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls") pod "success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" (UID: "78b47d4c-aba1-41e4-ab30-fed27babeb3b") : secret "success-200-isvc-fc1de-predictor-serving-cert" not found
Jul 28 12:57:46.025713 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.025692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.036036 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.036007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjtt\" (UniqueName: \"kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.529028 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.528982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.531550 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.531515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") pod \"success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") " pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.639289 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.639254    2560 generic.go:356] "Generic (PLEG): container finished" podID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerID="180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd" exitCode=2
Jul 28 12:57:46.639472 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.639321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerDied","Data":"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"}
Jul 28 12:57:46.688469 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.688428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:46.823821 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:46.823797    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:57:46.825928 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:57:46.825901    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b47d4c_aba1_41e4_ab30_fed27babeb3b.slice/crio-0616dbac2378c8ad33e47a4826e67051749cecb296ce9747957a50bd7e3b1483 WatchSource:0}: Error finding container 0616dbac2378c8ad33e47a4826e67051749cecb296ce9747957a50bd7e3b1483: Status 404 returned error can't find the container with id 0616dbac2378c8ad33e47a4826e67051749cecb296ce9747957a50bd7e3b1483
Jul 28 12:57:47.646681 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.646642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerStarted","Data":"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"}
Jul 28 12:57:47.646919 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.646709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerStarted","Data":"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"}
Jul 28 12:57:47.646919 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.646817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:47.646919 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.646841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerStarted","Data":"0616dbac2378c8ad33e47a4826e67051749cecb296ce9747957a50bd7e3b1483"}
Jul 28 12:57:47.647104 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.646984    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:47.648463 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.648435    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:57:47.666739 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:47.666679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podStartSLOduration=2.666663191 podStartE2EDuration="2.666663191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:57:47.665201745 +0000 UTC m=+1357.499946922" watchObservedRunningTime="2026-07-28 12:57:47.666663191 +0000 UTC m=+1357.501408366"
Jul 28 12:57:48.651284 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:48.651241    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:57:48.946286 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:48.946258    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:57:49.052498 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.052467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config\") pod \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") "
Jul 28 12:57:49.052695 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.052549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") pod \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") "
Jul 28 12:57:49.052695 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.052625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wms6q\" (UniqueName: \"kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q\") pod \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\" (UID: \"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c\") "
Jul 28 12:57:49.052904 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.052881    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" (UID: "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:49.054618 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.054587    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" (UID: "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:49.054726 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.054649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q" (OuterVolumeSpecName: "kube-api-access-wms6q") pod "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" (UID: "7bc0fda4-9a4b-42c3-adb8-d78a4d95615c"). InnerVolumeSpecName "kube-api-access-wms6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:57:49.153749 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.153708    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wms6q\" (UniqueName: \"kubernetes.io/projected/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-api-access-wms6q\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.153749 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.153741    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.153749 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.153751    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:57:49.657503 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.657463    2560 generic.go:356] "Generic (PLEG): container finished" podID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerID="b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72" exitCode=0
Jul 28 12:57:49.657503 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.657510    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"
Jul 28 12:57:49.658038 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.657554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerDied","Data":"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"}
Jul 28 12:57:49.658038 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.657625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6" event={"ID":"7bc0fda4-9a4b-42c3-adb8-d78a4d95615c","Type":"ContainerDied","Data":"0ffea86a20eace6c347d4c91c399af968c48f1f38ba5c42ac950440d5158fd7c"}
Jul 28 12:57:49.658038 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.657660    2560 scope.go:117] "RemoveContainer" containerID="180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"
Jul 28 12:57:49.667288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.667264    2560 scope.go:117] "RemoveContainer" containerID="b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"
Jul 28 12:57:49.675780 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.675761    2560 scope.go:117] "RemoveContainer" containerID="180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"
Jul 28 12:57:49.676113 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:57:49.676092    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd\": container with ID starting with 180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd not found: ID does not exist" containerID="180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"
Jul 28 12:57:49.676163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.676124    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd"} err="failed to get container status \"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd\": rpc error: code = NotFound desc = could not find container \"180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd\": container with ID starting with 180e750fe59ef4ad368f4228b34e6c50e94755e7fb639226390deab2f89ec9cd not found: ID does not exist"
Jul 28 12:57:49.676163 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.676144    2560 scope.go:117] "RemoveContainer" containerID="b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"
Jul 28 12:57:49.676408 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:57:49.676388    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72\": container with ID starting with b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72 not found: ID does not exist" containerID="b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"
Jul 28 12:57:49.676443 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.676416    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72"} err="failed to get container status \"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72\": rpc error: code = NotFound desc = could not find container \"b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72\": container with ID starting with b4fa8c4113c012836387deb03ce4ea9f0c8f213700945f5e69b9b08020b2cf72 not found: ID does not exist"
Jul 28 12:57:49.680536 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.680505    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:57:49.684351 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:49.684322    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7fa9-predictor-7fc8bd66b9-xgfn6"]
Jul 28 12:57:50.666665 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:50.666625    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" path="/var/lib/kubelet/pods/7bc0fda4-9a4b-42c3-adb8-d78a4d95615c/volumes"
Jul 28 12:57:53.656051 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:53.656018    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:57:53.656809 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:57:53.656781    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:58:03.657823 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:03.657780    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:58:06.206179 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.206137    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:58:06.206675 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.206401    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container" containerID="cri-o://38763508d405b2eb01ea5a604bb034b7f112d02ef3960c7b6a9cab8f3dd4899e" gracePeriod=30
Jul 28 12:58:06.206675 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.206466    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kube-rbac-proxy" containerID="cri-o://8b28bcd10e3c6f848ce46f9f4473a35bb99a4eac4429c31b9918e03e4e9e7237" gracePeriod=30
Jul 28 12:58:06.264493 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264459    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 12:58:06.264797 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264786    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kube-rbac-proxy"
Jul 28 12:58:06.264838 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264799    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kube-rbac-proxy"
Jul 28 12:58:06.264838 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264811    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container"
Jul 28 12:58:06.264838 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264816    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container"
Jul 28 12:58:06.264956 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264916    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kserve-container"
Jul 28 12:58:06.264956 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.264926    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc0fda4-9a4b-42c3-adb8-d78a4d95615c" containerName="kube-rbac-proxy"
Jul 28 12:58:06.269614 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.269588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.272262 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.272234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-da172-kube-rbac-proxy-sar-config\""
Jul 28 12:58:06.272775 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.272682    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-da172-predictor-serving-cert\""
Jul 28 12:58:06.278472 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.278439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 12:58:06.298112 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.298068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7dc\" (UniqueName: \"kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.298288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.298143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.298288 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.298214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.398637 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.398598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.398822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.398657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7dc\" (UniqueName: \"kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.398822 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.398708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.398822 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:06.398821    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-da172-predictor-serving-cert: secret "success-200-isvc-da172-predictor-serving-cert" not found
Jul 28 12:58:06.399060 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:06.398907    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls podName:ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966 nodeName:}" failed. No retries permitted until 2026-07-28 12:58:06.898885217 +0000 UTC m=+1376.733630384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls") pod "success-200-isvc-da172-predictor-79f56c75f7-5c7vk" (UID: "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966") : secret "success-200-isvc-da172-predictor-serving-cert" not found
Jul 28 12:58:06.399810 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.399783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.410027 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.409992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7dc\" (UniqueName: \"kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.733395 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.733362    2560 generic.go:356] "Generic (PLEG): container finished" podID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerID="8b28bcd10e3c6f848ce46f9f4473a35bb99a4eac4429c31b9918e03e4e9e7237" exitCode=2
Jul 28 12:58:06.733583 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.733436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerDied","Data":"8b28bcd10e3c6f848ce46f9f4473a35bb99a4eac4429c31b9918e03e4e9e7237"}
Jul 28 12:58:06.903036 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.902998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:06.905557 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:06.905533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") pod \"success-200-isvc-da172-predictor-79f56c75f7-5c7vk\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") " pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:07.182093 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.181989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:07.332032 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.331841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 12:58:07.334626 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:58:07.334597    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8ca64f_9e38_4fd4_8e4b_dcef9bcef966.slice/crio-1018a5a1cf18fa11040651880ede31c14a9ad6fce646a0f411ea2b63c8b72dca WatchSource:0}: Error finding container 1018a5a1cf18fa11040651880ede31c14a9ad6fce646a0f411ea2b63c8b72dca: Status 404 returned error can't find the container with id 1018a5a1cf18fa11040651880ede31c14a9ad6fce646a0f411ea2b63c8b72dca
Jul 28 12:58:07.749505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.749043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerStarted","Data":"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"}
Jul 28 12:58:07.749505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.749170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerStarted","Data":"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"}
Jul 28 12:58:07.749505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.749203    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerStarted","Data":"1018a5a1cf18fa11040651880ede31c14a9ad6fce646a0f411ea2b63c8b72dca"}
Jul 28 12:58:07.749505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.749413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:07.749505 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.749488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:07.751156 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.751125    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:07.769034 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:07.768977    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podStartSLOduration=1.768961353 podStartE2EDuration="1.768961353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:07.76825355 +0000 UTC m=+1377.602998753" watchObservedRunningTime="2026-07-28 12:58:07.768961353 +0000 UTC m=+1377.603706527"
Jul 28 12:58:08.429023 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:08.428974    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 12:58:08.754656 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:08.754605    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:09.760470 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.760428    2560 generic.go:356] "Generic (PLEG): container finished" podID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerID="38763508d405b2eb01ea5a604bb034b7f112d02ef3960c7b6a9cab8f3dd4899e" exitCode=0
Jul 28 12:58:09.760841 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.760476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerDied","Data":"38763508d405b2eb01ea5a604bb034b7f112d02ef3960c7b6a9cab8f3dd4899e"}
Jul 28 12:58:09.872859 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.872830    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:58:09.928984 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.928883    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config\") pod \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") "
Jul 28 12:58:09.928984 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.928961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22xj2\" (UniqueName: \"kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2\") pod \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") "
Jul 28 12:58:09.929191 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.928996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") pod \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\" (UID: \"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9\") "
Jul 28 12:58:09.929257 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.929231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" (UID: "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:09.931060 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.931024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" (UID: "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:09.931171 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:09.931112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2" (OuterVolumeSpecName: "kube-api-access-22xj2") pod "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" (UID: "6c748ae6-3fed-4d4e-8cc0-57fa282a85d9"). InnerVolumeSpecName "kube-api-access-22xj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:58:10.030113 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.030061    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:10.030113 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.030104    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22xj2\" (UniqueName: \"kubernetes.io/projected/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-kube-api-access-22xj2\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:10.030113 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.030119    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:10.766771 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.766729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8" event={"ID":"6c748ae6-3fed-4d4e-8cc0-57fa282a85d9","Type":"ContainerDied","Data":"6e7d72b2dff030c6ec123cf450a84f4aa0efc94afc6e4a89392c8406ceaa1fcd"}
Jul 28 12:58:10.767236 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.766806    2560 scope.go:117] "RemoveContainer" containerID="8b28bcd10e3c6f848ce46f9f4473a35bb99a4eac4429c31b9918e03e4e9e7237"
Jul 28 12:58:10.767236 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.766751    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"
Jul 28 12:58:10.775176 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.775154    2560 scope.go:117] "RemoveContainer" containerID="38763508d405b2eb01ea5a604bb034b7f112d02ef3960c7b6a9cab8f3dd4899e"
Jul 28 12:58:10.787753 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.787723    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:58:10.792274 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:10.792242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-75e21-predictor-7df67ccd87-mnfc8"]
Jul 28 12:58:12.661744 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:12.661703    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" path="/var/lib/kubelet/pods/6c748ae6-3fed-4d4e-8cc0-57fa282a85d9/volumes"
Jul 28 12:58:13.657429 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:13.657389    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:58:13.759323 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:13.759287    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:13.759796 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:13.759768    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:23.656998 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:23.656953    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:58:23.760708 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:23.760664    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:33.657806 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:33.657769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:58:33.760546 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:33.760508    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:43.760499 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:43.760452    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 12:58:53.760735 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:53.760695    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 12:58:55.994275 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:55.994241    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:58:55.994845 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:55.994612    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container" containerID="cri-o://64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a" gracePeriod=30
Jul 28 12:58:55.994845 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:55.994622    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kube-rbac-proxy" containerID="cri-o://4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201" gracePeriod=30
Jul 28 12:58:56.070504 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070468    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 12:58:56.070824 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070812    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container"
Jul 28 12:58:56.070883 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070825    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container"
Jul 28 12:58:56.070883 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070842    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kube-rbac-proxy"
Jul 28 12:58:56.070883 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070847    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kube-rbac-proxy"
Jul 28 12:58:56.070984 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070914    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kube-rbac-proxy"
Jul 28 12:58:56.070984 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.070927    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c748ae6-3fed-4d4e-8cc0-57fa282a85d9" containerName="kserve-container"
Jul 28 12:58:56.074443 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.074416    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.077229 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.077200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d01ec-predictor-serving-cert\""
Jul 28 12:58:56.077400 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.077206    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d01ec-kube-rbac-proxy-sar-config\""
Jul 28 12:58:56.090254 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.090219    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 12:58:56.140963 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.140909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngc6s\" (UniqueName: \"kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.141151 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.141081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.141151 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.141124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.241771 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.241725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngc6s\" (UniqueName: \"kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.242005 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.241882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.242005 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.241918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.242150 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:56.242051    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-serving-cert: secret "success-200-isvc-d01ec-predictor-serving-cert" not found
Jul 28 12:58:56.242150 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:56.242114    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls podName:22af2408-38a6-4630-9afe-678a75a9954d nodeName:}" failed. No retries permitted until 2026-07-28 12:58:56.742095284 +0000 UTC m=+1426.576840437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls") pod "success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" (UID: "22af2408-38a6-4630-9afe-678a75a9954d") : secret "success-200-isvc-d01ec-predictor-serving-cert" not found
Jul 28 12:58:56.242691 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.242663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.253679 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.253599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngc6s\" (UniqueName: \"kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.744643 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.744594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:56.744831 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:56.744767    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-serving-cert: secret "success-200-isvc-d01ec-predictor-serving-cert" not found
Jul 28 12:58:56.744890 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:58:56.744851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls podName:22af2408-38a6-4630-9afe-678a75a9954d nodeName:}" failed. No retries permitted until 2026-07-28 12:58:57.744830901 +0000 UTC m=+1427.579576069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls") pod "success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" (UID: "22af2408-38a6-4630-9afe-678a75a9954d") : secret "success-200-isvc-d01ec-predictor-serving-cert" not found
Jul 28 12:58:56.964874 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.964819    2560 generic.go:356] "Generic (PLEG): container finished" podID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerID="4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201" exitCode=2
Jul 28 12:58:56.965037 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:56.964902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerDied","Data":"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"}
Jul 28 12:58:57.752054 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:57.752012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:57.754573 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:57.754547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"success-200-isvc-d01ec-predictor-6667644fcc-v6sbh\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") " pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:57.885997 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:57.885958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:58.027668 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.027635    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 12:58:58.030366 ip-10-0-134-201 kubenswrapper[2560]: W0728 12:58:58.030326    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22af2408_38a6_4630_9afe_678a75a9954d.slice/crio-c9f85c458db8f9127745995e15c8340bb092b0e74c8f7ad45addfdc540f8a3b9 WatchSource:0}: Error finding container c9f85c458db8f9127745995e15c8340bb092b0e74c8f7ad45addfdc540f8a3b9: Status 404 returned error can't find the container with id c9f85c458db8f9127745995e15c8340bb092b0e74c8f7ad45addfdc540f8a3b9
Jul 28 12:58:58.651551 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.651502    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 12:58:58.975754 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.975716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerStarted","Data":"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"}
Jul 28 12:58:58.976196 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.975777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerStarted","Data":"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"}
Jul 28 12:58:58.976196 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.975802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:58.976196 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.975822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerStarted","Data":"c9f85c458db8f9127745995e15c8340bb092b0e74c8f7ad45addfdc540f8a3b9"}
Jul 28 12:58:58.995424 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:58.995360    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podStartSLOduration=2.9953357819999997 podStartE2EDuration="2.995335782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:58.993454015 +0000 UTC m=+1428.828199251" watchObservedRunningTime="2026-07-28 12:58:58.995335782 +0000 UTC m=+1428.830080957"
Jul 28 12:58:59.757201 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.757174    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:58:59.771622 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.771579    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmjtt\" (UniqueName: \"kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt\") pod \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") "
Jul 28 12:58:59.771808 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.771631    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") pod \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") "
Jul 28 12:58:59.771808 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.771670    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config\") pod \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\" (UID: \"78b47d4c-aba1-41e4-ab30-fed27babeb3b\") "
Jul 28 12:58:59.772148 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.772124    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78b47d4c-aba1-41e4-ab30-fed27babeb3b" (UID: "78b47d4c-aba1-41e4-ab30-fed27babeb3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:59.773815 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.773790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78b47d4c-aba1-41e4-ab30-fed27babeb3b" (UID: "78b47d4c-aba1-41e4-ab30-fed27babeb3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:59.773908 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.773842    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt" (OuterVolumeSpecName: "kube-api-access-rmjtt") pod "78b47d4c-aba1-41e4-ab30-fed27babeb3b" (UID: "78b47d4c-aba1-41e4-ab30-fed27babeb3b"). InnerVolumeSpecName "kube-api-access-rmjtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:58:59.873142 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.873053    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78b47d4c-aba1-41e4-ab30-fed27babeb3b-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.873142 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.873081    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.873142 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.873092    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjtt\" (UniqueName: \"kubernetes.io/projected/78b47d4c-aba1-41e4-ab30-fed27babeb3b-kube-api-access-rmjtt\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 12:58:59.982210 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982178    2560 generic.go:356] "Generic (PLEG): container finished" podID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerID="64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a" exitCode=0
Jul 28 12:58:59.982626 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982226    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"
Jul 28 12:58:59.982626 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982267    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerDied","Data":"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"}
Jul 28 12:58:59.982626 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6" event={"ID":"78b47d4c-aba1-41e4-ab30-fed27babeb3b","Type":"ContainerDied","Data":"0616dbac2378c8ad33e47a4826e67051749cecb296ce9747957a50bd7e3b1483"}
Jul 28 12:58:59.982626 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982374    2560 scope.go:117] "RemoveContainer" containerID="4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"
Jul 28 12:58:59.982840 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.982735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:58:59.984452 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.984423    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:58:59.992338 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:58:59.992314    2560 scope.go:117] "RemoveContainer" containerID="64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"
Jul 28 12:59:00.000329 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.000304    2560 scope.go:117] "RemoveContainer" containerID="4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"
Jul 28 12:59:00.000628 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:59:00.000605    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201\": container with ID starting with 4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201 not found: ID does not exist" containerID="4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"
Jul 28 12:59:00.000701 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.000644    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201"} err="failed to get container status \"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201\": rpc error: code = NotFound desc = could not find container \"4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201\": container with ID starting with 4772a54ca7a3a2956d4201806d378e4dbfb4ff5b2b23c0f766c1a61099eaa201 not found: ID does not exist"
Jul 28 12:59:00.000701 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.000669    2560 scope.go:117] "RemoveContainer" containerID="64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"
Jul 28 12:59:00.000977 ip-10-0-134-201 kubenswrapper[2560]: E0728 12:59:00.000959    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a\": container with ID starting with 64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a not found: ID does not exist" containerID="64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"
Jul 28 12:59:00.001024 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.000986    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a"} err="failed to get container status \"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a\": rpc error: code = NotFound desc = could not find container \"64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a\": container with ID starting with 64475fe68a284e60b758fb82f80cf56841f11bef3bbcc1b458aad018d0f1511a not found: ID does not exist"
Jul 28 12:59:00.009063 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.009032    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:59:00.013618 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.013589    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc1de-predictor-fb95f5f9c-nwhh6"]
Jul 28 12:59:00.662018 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.661987    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" path="/var/lib/kubelet/pods/78b47d4c-aba1-41e4-ab30-fed27babeb3b/volumes"
Jul 28 12:59:00.987737 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:00.987695    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:05.991940 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:05.991907    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 12:59:05.992511 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:05.992482    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:15.992676 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:15.992636    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:25.992616 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:25.992569    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:35.992916 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:35.992854    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 12:59:45.993777 ip-10-0-134-201 kubenswrapper[2560]: I0728 12:59:45.993743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 13:00:10.645089 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:00:10.645056    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:00:10.646702 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:00:10.646678    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:05:10.673883 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:05:10.673843    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:05:10.675888 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:05:10.675841    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:07:21.133493 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.133450    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 13:07:21.134684 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.133884    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kube-rbac-proxy" containerID="cri-o://0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c" gracePeriod=30
Jul 28 13:07:21.134684 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.133840    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" containerID="cri-o://fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e" gracePeriod=30
Jul 28 13:07:21.182969 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.182928    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:07:21.183286 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183271    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container"
Jul 28 13:07:21.183361 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183287    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container"
Jul 28 13:07:21.183361 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183302    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kube-rbac-proxy"
Jul 28 13:07:21.183361 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183308    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kube-rbac-proxy"
Jul 28 13:07:21.183508 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183372    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kserve-container"
Jul 28 13:07:21.183508 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.183381    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b47d4c-aba1-41e4-ab30-fed27babeb3b" containerName="kube-rbac-proxy"
Jul 28 13:07:21.186501 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.186476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.189114 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.189089    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b3a0-predictor-serving-cert\""
Jul 28 13:07:21.189114 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.189109    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4b3a0-kube-rbac-proxy-sar-config\""
Jul 28 13:07:21.195940 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.195534    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:07:21.281967 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.281698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.281967 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.281781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.281967 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.281822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j2k\" (UniqueName: \"kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.382770 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.382724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.382982 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.382791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.382982 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.382822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j2k\" (UniqueName: \"kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.382982 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:07:21.382964    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-serving-cert: secret "success-200-isvc-4b3a0-predictor-serving-cert" not found
Jul 28 13:07:21.383152 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:07:21.383039    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls podName:dedeb632-087e-4ced-bce7-d8c8a770fecb nodeName:}" failed. No retries permitted until 2026-07-28 13:07:21.883015795 +0000 UTC m=+1931.717760967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls") pod "success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" (UID: "dedeb632-087e-4ced-bce7-d8c8a770fecb") : secret "success-200-isvc-4b3a0-predictor-serving-cert" not found
Jul 28 13:07:21.383450 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.383430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.395126 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.395038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j2k\" (UniqueName: \"kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.888039 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.888001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:21.890565 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:21.890524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") pod \"success-200-isvc-4b3a0-predictor-f958c8967-5ttpg\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") " pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:22.063938 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:22.063898    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerID="0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c" exitCode=2
Jul 28 13:07:22.064103 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:22.063953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerDied","Data":"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"}
Jul 28 13:07:22.102254 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:22.102223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:22.239581 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:22.239553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:07:22.242291 ip-10-0-134-201 kubenswrapper[2560]: W0728 13:07:22.242251    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedeb632_087e_4ced_bce7_d8c8a770fecb.slice/crio-45203b6986a587d2da6643097af40c7bf1b29bc773cf3f4d811fb09d8d1d8cd3 WatchSource:0}: Error finding container 45203b6986a587d2da6643097af40c7bf1b29bc773cf3f4d811fb09d8d1d8cd3: Status 404 returned error can't find the container with id 45203b6986a587d2da6643097af40c7bf1b29bc773cf3f4d811fb09d8d1d8cd3
Jul 28 13:07:23.069155 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.069117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerStarted","Data":"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"}
Jul 28 13:07:23.069336 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.069185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerStarted","Data":"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"}
Jul 28 13:07:23.069336 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.069207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerStarted","Data":"45203b6986a587d2da6643097af40c7bf1b29bc773cf3f4d811fb09d8d1d8cd3"}
Jul 28 13:07:23.069415 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.069349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:23.069526 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.069514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:23.070780 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.070758    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:07:23.093172 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.093114    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podStartSLOduration=2.093093702 podStartE2EDuration="2.093093702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:07:23.092162914 +0000 UTC m=+1932.926908115" watchObservedRunningTime="2026-07-28 13:07:23.093093702 +0000 UTC m=+1932.927838876"
Jul 28 13:07:23.755674 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.755626    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 13:07:23.759996 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:23.759969    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:07:24.080733 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.080637    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:07:24.688838 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.688807    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 13:07:24.816094 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.815998    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv7dc\" (UniqueName: \"kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc\") pod \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") "
Jul 28 13:07:24.816094 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.816062    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config\") pod \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") "
Jul 28 13:07:24.816818 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.816127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") pod \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\" (UID: \"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966\") "
Jul 28 13:07:24.816818 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.816383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" (UID: "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:07:24.818318 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.818294    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" (UID: "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:07:24.818418 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.818313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc" (OuterVolumeSpecName: "kube-api-access-tv7dc") pod "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" (UID: "ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966"). InnerVolumeSpecName "kube-api-access-tv7dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:07:24.916959 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.916918    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv7dc\" (UniqueName: \"kubernetes.io/projected/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-api-access-tv7dc\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:07:24.916959 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.916953    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:07:24.916959 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:24.916963    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:07:25.086043 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.085947    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerID="fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e" exitCode=0
Jul 28 13:07:25.086043 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.085997    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"
Jul 28 13:07:25.086259 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.086037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerDied","Data":"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"}
Jul 28 13:07:25.086259 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.086100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk" event={"ID":"ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966","Type":"ContainerDied","Data":"1018a5a1cf18fa11040651880ede31c14a9ad6fce646a0f411ea2b63c8b72dca"}
Jul 28 13:07:25.086259 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.086127    2560 scope.go:117] "RemoveContainer" containerID="0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"
Jul 28 13:07:25.094848 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.094817    2560 scope.go:117] "RemoveContainer" containerID="fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"
Jul 28 13:07:25.103120 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.103096    2560 scope.go:117] "RemoveContainer" containerID="0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"
Jul 28 13:07:25.103431 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:07:25.103410    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c\": container with ID starting with 0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c not found: ID does not exist" containerID="0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"
Jul 28 13:07:25.103511 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.103441    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c"} err="failed to get container status \"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c\": rpc error: code = NotFound desc = could not find container \"0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c\": container with ID starting with 0e9d5b2f763ee107786213744f4a1e1e5548d8c72cdb94610f5a6e9da3ea874c not found: ID does not exist"
Jul 28 13:07:25.103511 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.103460    2560 scope.go:117] "RemoveContainer" containerID="fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"
Jul 28 13:07:25.103733 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:07:25.103718    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e\": container with ID starting with fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e not found: ID does not exist" containerID="fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"
Jul 28 13:07:25.103777 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.103737    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e"} err="failed to get container status \"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e\": rpc error: code = NotFound desc = could not find container \"fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e\": container with ID starting with fb425aed74b35afbb23591e6257f5fe729f2d48b4f1bc09fdabbed7b9c56bc8e not found: ID does not exist"
Jul 28 13:07:25.108620 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.108590    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 13:07:25.113423 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:25.113396    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-da172-predictor-79f56c75f7-5c7vk"]
Jul 28 13:07:26.661934 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:26.661897    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" path="/var/lib/kubelet/pods/ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966/volumes"
Jul 28 13:07:29.084938 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:29.084907    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:07:29.085381 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:29.085351    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:07:39.085468 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:39.085416    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:07:49.086334 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:49.086286    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:07:59.085977 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:07:59.085933    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:08:09.086954 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:09.086845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:08:20.835082 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.835040    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 13:08:20.835584 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.835344    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" containerID="cri-o://80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0" gracePeriod=30
Jul 28 13:08:20.835798 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.835771    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy" containerID="cri-o://05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95" gracePeriod=30
Jul 28 13:08:20.925121 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925080    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:08:20.925544 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925522    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kube-rbac-proxy"
Jul 28 13:08:20.925544 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925544    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kube-rbac-proxy"
Jul 28 13:08:20.925704 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925555    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container"
Jul 28 13:08:20.925704 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925561    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container"
Jul 28 13:08:20.925704 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925643    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kserve-container"
Jul 28 13:08:20.925704 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.925653    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea8ca64f-9e38-4fd4-8e4b-dcef9bcef966" containerName="kube-rbac-proxy"
Jul 28 13:08:20.928917 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.928891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:20.931602 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.931571    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dde97-predictor-serving-cert\""
Jul 28 13:08:20.931765 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.931746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dde97-kube-rbac-proxy-sar-config\""
Jul 28 13:08:20.938451 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.938422    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:08:20.988480 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:20.988435    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 13:08:21.025258 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.025216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.025439 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.025370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.025439 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.025402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdsk\" (UniqueName: \"kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.125924 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.125811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.125924 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.125848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdsk\" (UniqueName: \"kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.125924 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.125888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.126773 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.126737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.128915 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.128889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.135047 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.135003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdsk\" (UniqueName: \"kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk\") pod \"success-200-isvc-dde97-predictor-8465f8b49c-h25g6\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") " pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.240688 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.240645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:21.323194 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.323151    2560 generic.go:356] "Generic (PLEG): container finished" podID="22af2408-38a6-4630-9afe-678a75a9954d" containerID="05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95" exitCode=2
Jul 28 13:08:21.323358 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.323285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerDied","Data":"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"}
Jul 28 13:08:21.388442 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:21.388412    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:08:21.391079 ip-10-0-134-201 kubenswrapper[2560]: W0728 13:08:21.391041    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a107a6_dd9a_4044_8da6_6b8013f60ad8.slice/crio-64afd7380b0c791cbe0e291cc1b4c13c12ae03f9803272a745f98d803f87c7dd WatchSource:0}: Error finding container 64afd7380b0c791cbe0e291cc1b4c13c12ae03f9803272a745f98d803f87c7dd: Status 404 returned error can't find the container with id 64afd7380b0c791cbe0e291cc1b4c13c12ae03f9803272a745f98d803f87c7dd
Jul 28 13:08:22.329789 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.329745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerStarted","Data":"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"}
Jul 28 13:08:22.330205 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.329803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerStarted","Data":"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"}
Jul 28 13:08:22.330205 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.329826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerStarted","Data":"64afd7380b0c791cbe0e291cc1b4c13c12ae03f9803272a745f98d803f87c7dd"}
Jul 28 13:08:22.330205 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.329851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:22.330205 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.329979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:22.331520 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.331491    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:22.346947 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:22.346885    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podStartSLOduration=2.346845623 podStartE2EDuration="2.346845623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:22.346143904 +0000 UTC m=+1992.180889090" watchObservedRunningTime="2026-07-28 13:08:22.346845623 +0000 UTC m=+1992.181590798"
Jul 28 13:08:23.333771 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:23.333734    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:25.988170 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:25.988124    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 13:08:25.993492 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:25.993464    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 13:08:27.684236 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.684207    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 13:08:27.776405 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.776300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngc6s\" (UniqueName: \"kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s\") pod \"22af2408-38a6-4630-9afe-678a75a9954d\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") "
Jul 28 13:08:27.776405 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.776364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") pod \"22af2408-38a6-4630-9afe-678a75a9954d\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") "
Jul 28 13:08:27.776642 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.776416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config\") pod \"22af2408-38a6-4630-9afe-678a75a9954d\" (UID: \"22af2408-38a6-4630-9afe-678a75a9954d\") "
Jul 28 13:08:27.776794 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.776761    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22af2408-38a6-4630-9afe-678a75a9954d" (UID: "22af2408-38a6-4630-9afe-678a75a9954d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:27.778486 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.778463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s" (OuterVolumeSpecName: "kube-api-access-ngc6s") pod "22af2408-38a6-4630-9afe-678a75a9954d" (UID: "22af2408-38a6-4630-9afe-678a75a9954d"). InnerVolumeSpecName "kube-api-access-ngc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:27.778638 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.778615    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22af2408-38a6-4630-9afe-678a75a9954d" (UID: "22af2408-38a6-4630-9afe-678a75a9954d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:27.877138 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.877105    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngc6s\" (UniqueName: \"kubernetes.io/projected/22af2408-38a6-4630-9afe-678a75a9954d-kube-api-access-ngc6s\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:27.877138 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.877132    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22af2408-38a6-4630-9afe-678a75a9954d-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:27.877138 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:27.877142    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22af2408-38a6-4630-9afe-678a75a9954d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:28.338958 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.338924    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:08:28.339402 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.339369    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:28.356581 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.356546    2560 generic.go:356] "Generic (PLEG): container finished" podID="22af2408-38a6-4630-9afe-678a75a9954d" containerID="80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0" exitCode=0
Jul 28 13:08:28.356759 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.356603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"
Jul 28 13:08:28.356759 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.356624    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerDied","Data":"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"}
Jul 28 13:08:28.356759 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.356686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh" event={"ID":"22af2408-38a6-4630-9afe-678a75a9954d","Type":"ContainerDied","Data":"c9f85c458db8f9127745995e15c8340bb092b0e74c8f7ad45addfdc540f8a3b9"}
Jul 28 13:08:28.356759 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.356714    2560 scope.go:117] "RemoveContainer" containerID="05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"
Jul 28 13:08:28.366564 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.366538    2560 scope.go:117] "RemoveContainer" containerID="80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"
Jul 28 13:08:28.374441 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.374420    2560 scope.go:117] "RemoveContainer" containerID="05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"
Jul 28 13:08:28.374758 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:08:28.374735    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95\": container with ID starting with 05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95 not found: ID does not exist" containerID="05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"
Jul 28 13:08:28.374797 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.374769    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95"} err="failed to get container status \"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95\": rpc error: code = NotFound desc = could not find container \"05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95\": container with ID starting with 05f0ebbdff07ddeaa95d6f4d569c88cae2e8d53ef212744ec930577a41d20d95 not found: ID does not exist"
Jul 28 13:08:28.374797 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.374788    2560 scope.go:117] "RemoveContainer" containerID="80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"
Jul 28 13:08:28.375067 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:08:28.375049    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0\": container with ID starting with 80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0 not found: ID does not exist" containerID="80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"
Jul 28 13:08:28.375113 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.375074    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0"} err="failed to get container status \"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0\": rpc error: code = NotFound desc = could not find container \"80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0\": container with ID starting with 80ee655f0527501bd86b0412557d89702a37604a12ff163ccb7fc9a62c80ebd0 not found: ID does not exist"
Jul 28 13:08:28.379984 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.379960    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 13:08:28.383533 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.383510    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d01ec-predictor-6667644fcc-v6sbh"]
Jul 28 13:08:28.661324 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:28.661238    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22af2408-38a6-4630-9afe-678a75a9954d" path="/var/lib/kubelet/pods/22af2408-38a6-4630-9afe-678a75a9954d/volumes"
Jul 28 13:08:31.410969 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.410891    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:08:31.411463 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.411121    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container" containerID="cri-o://ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70" gracePeriod=30
Jul 28 13:08:31.411463 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.411166    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kube-rbac-proxy" containerID="cri-o://530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664" gracePeriod=30
Jul 28 13:08:31.482185 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482072    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:08:31.482599 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482579    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container"
Jul 28 13:08:31.482599 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482602    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container"
Jul 28 13:08:31.482758 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482614    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy"
Jul 28 13:08:31.482758 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482622    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy"
Jul 28 13:08:31.482758 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482735    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kserve-container"
Jul 28 13:08:31.482758 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.482747    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22af2408-38a6-4630-9afe-678a75a9954d" containerName="kube-rbac-proxy"
Jul 28 13:08:31.487499 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.487444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.490272 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.490242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-13bf1-kube-rbac-proxy-sar-config\""
Jul 28 13:08:31.490430 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.490385    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-13bf1-predictor-serving-cert\""
Jul 28 13:08:31.495631 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.495420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:08:31.507939 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.507896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.507939 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.507937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdr4\" (UniqueName: \"kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.508199 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.508007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.609226 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.609187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.609422 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.609264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.609422 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.609297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdr4\" (UniqueName: \"kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.610081 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.610055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.612066 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.612035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.617530 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.617499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdr4\" (UniqueName: \"kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4\") pod \"success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") " pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.800346 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.800298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:31.941524 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:31.941491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:08:31.944256 ip-10-0-134-201 kubenswrapper[2560]: W0728 13:08:31.944221    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53ddbe5_6d13_4fa0_9eb9_0f5f3847b280.slice/crio-1917f0274ff66ee9308af5716d85a3ab68b7b309cebae6ea46259d7703596e5c WatchSource:0}: Error finding container 1917f0274ff66ee9308af5716d85a3ab68b7b309cebae6ea46259d7703596e5c: Status 404 returned error can't find the container with id 1917f0274ff66ee9308af5716d85a3ab68b7b309cebae6ea46259d7703596e5c
Jul 28 13:08:32.377448 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.377331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerStarted","Data":"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"}
Jul 28 13:08:32.377448 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.377396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerStarted","Data":"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"}
Jul 28 13:08:32.377693 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.377519    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:32.377693 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.377558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerStarted","Data":"1917f0274ff66ee9308af5716d85a3ab68b7b309cebae6ea46259d7703596e5c"}
Jul 28 13:08:32.379091 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.379064    2560 generic.go:356] "Generic (PLEG): container finished" podID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerID="530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664" exitCode=2
Jul 28 13:08:32.379225 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.379133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerDied","Data":"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"}
Jul 28 13:08:32.396331 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:32.396271    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podStartSLOduration=1.396251532 podStartE2EDuration="1.396251532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:08:32.394788906 +0000 UTC m=+2002.229534110" watchObservedRunningTime="2026-07-28 13:08:32.396251532 +0000 UTC m=+2002.230996707"
Jul 28 13:08:33.384444 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:33.384398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:33.385807 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:33.385771    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:08:34.080850 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:34.080804    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:08:34.389052 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:34.388949    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:08:35.369480 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.369445    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:08:35.396245 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.396161    2560 generic.go:356] "Generic (PLEG): container finished" podID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerID="ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70" exitCode=0
Jul 28 13:08:35.396245 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.396221    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"
Jul 28 13:08:35.396245 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.396238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerDied","Data":"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"}
Jul 28 13:08:35.396738 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.396303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg" event={"ID":"dedeb632-087e-4ced-bce7-d8c8a770fecb","Type":"ContainerDied","Data":"45203b6986a587d2da6643097af40c7bf1b29bc773cf3f4d811fb09d8d1d8cd3"}
Jul 28 13:08:35.396738 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.396329    2560 scope.go:117] "RemoveContainer" containerID="530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"
Jul 28 13:08:35.405071 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.405050    2560 scope.go:117] "RemoveContainer" containerID="ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"
Jul 28 13:08:35.414751 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.414729    2560 scope.go:117] "RemoveContainer" containerID="530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"
Jul 28 13:08:35.415144 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:08:35.415121    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664\": container with ID starting with 530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664 not found: ID does not exist" containerID="530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"
Jul 28 13:08:35.415207 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.415152    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664"} err="failed to get container status \"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664\": rpc error: code = NotFound desc = could not find container \"530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664\": container with ID starting with 530eef379b236c4617741de3bf7b812010740d207d218c6039aa78c4f4947664 not found: ID does not exist"
Jul 28 13:08:35.415207 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.415172    2560 scope.go:117] "RemoveContainer" containerID="ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"
Jul 28 13:08:35.415462 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:08:35.415445    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70\": container with ID starting with ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70 not found: ID does not exist" containerID="ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"
Jul 28 13:08:35.415512 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.415468    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70"} err="failed to get container status \"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70\": rpc error: code = NotFound desc = could not find container \"ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70\": container with ID starting with ed71286146e69e9c8ce38ade843df2af24e0a21db0bda8069f1a42a4c521cd70 not found: ID does not exist"
Jul 28 13:08:35.440896 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.440847    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6j2k\" (UniqueName: \"kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k\") pod \"dedeb632-087e-4ced-bce7-d8c8a770fecb\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") "
Jul 28 13:08:35.441102 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.440913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config\") pod \"dedeb632-087e-4ced-bce7-d8c8a770fecb\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") "
Jul 28 13:08:35.441102 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.441006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") pod \"dedeb632-087e-4ced-bce7-d8c8a770fecb\" (UID: \"dedeb632-087e-4ced-bce7-d8c8a770fecb\") "
Jul 28 13:08:35.441321 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.441294    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dedeb632-087e-4ced-bce7-d8c8a770fecb" (UID: "dedeb632-087e-4ced-bce7-d8c8a770fecb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:08:35.443092 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.443059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dedeb632-087e-4ced-bce7-d8c8a770fecb" (UID: "dedeb632-087e-4ced-bce7-d8c8a770fecb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:08:35.443213 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.443089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k" (OuterVolumeSpecName: "kube-api-access-m6j2k") pod "dedeb632-087e-4ced-bce7-d8c8a770fecb" (UID: "dedeb632-087e-4ced-bce7-d8c8a770fecb"). InnerVolumeSpecName "kube-api-access-m6j2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:08:35.542180 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.542142    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dedeb632-087e-4ced-bce7-d8c8a770fecb-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:35.542180 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.542174    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6j2k\" (UniqueName: \"kubernetes.io/projected/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-api-access-m6j2k\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:35.542180 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.542184    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dedeb632-087e-4ced-bce7-d8c8a770fecb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:08:35.723137 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.723092    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:08:35.729547 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:35.729510    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4b3a0-predictor-f958c8967-5ttpg"]
Jul 28 13:08:36.662021 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:36.661988    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" path="/var/lib/kubelet/pods/dedeb632-087e-4ced-bce7-d8c8a770fecb/volumes"
Jul 28 13:08:38.339839 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:38.339797    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:39.393814 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:39.393779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:08:39.394457 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:39.394428    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:08:48.339936 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:48.339883    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:49.395139 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:49.395096    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:08:58.339767 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:58.339726    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:08:59.395181 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:08:59.395127    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:09:08.340017 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:09:08.339983    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:09:09.394662 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:09:09.394622    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:09:19.395719 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:09:19.395684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:10:10.698757 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:10:10.698727    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:10:10.702072 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:10:10.702047    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:15:10.724387 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:15:10.724263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:15:10.728775 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:15:10.728753    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:17:46.396030 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:46.395993    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:17:46.396593 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:46.396227    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" containerID="cri-o://b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd" gracePeriod=30
Jul 28 13:17:46.396593 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:46.396280    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kube-rbac-proxy" containerID="cri-o://bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633" gracePeriod=30
Jul 28 13:17:46.675710 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:46.675605    2560 generic.go:356] "Generic (PLEG): container finished" podID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerID="bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633" exitCode=2
Jul 28 13:17:46.675710 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:46.675672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerDied","Data":"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"}
Jul 28 13:17:49.389352 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.389304    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 13:17:49.394788 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.394750    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:17:49.849134 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.849106    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:17:49.970080 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.970042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrdr4\" (UniqueName: \"kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4\") pod \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") "
Jul 28 13:17:49.970256 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.970118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls\") pod \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") "
Jul 28 13:17:49.970256 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.970166    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config\") pod \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\" (UID: \"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280\") "
Jul 28 13:17:49.970567 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.970538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" (UID: "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:17:49.972248 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.972219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" (UID: "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:17:49.972388 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:49.972332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4" (OuterVolumeSpecName: "kube-api-access-lrdr4") pod "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" (UID: "e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280"). InnerVolumeSpecName "kube-api-access-lrdr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:17:50.071103 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.071064    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:17:50.071103 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.071095    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdr4\" (UniqueName: \"kubernetes.io/projected/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-kube-api-access-lrdr4\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:17:50.071103 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.071104    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:17:50.693163 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.693128    2560 generic.go:356] "Generic (PLEG): container finished" podID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerID="b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd" exitCode=0
Jul 28 13:17:50.693582 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.693187    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"
Jul 28 13:17:50.693582 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.693217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerDied","Data":"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"}
Jul 28 13:17:50.693582 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.693277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m" event={"ID":"e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280","Type":"ContainerDied","Data":"1917f0274ff66ee9308af5716d85a3ab68b7b309cebae6ea46259d7703596e5c"}
Jul 28 13:17:50.693582 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.693304    2560 scope.go:117] "RemoveContainer" containerID="bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"
Jul 28 13:17:50.701810 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.701788    2560 scope.go:117] "RemoveContainer" containerID="b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"
Jul 28 13:17:50.709719 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.709691    2560 scope.go:117] "RemoveContainer" containerID="bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"
Jul 28 13:17:50.710293 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:17:50.710232    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633\": container with ID starting with bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633 not found: ID does not exist" containerID="bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"
Jul 28 13:17:50.710442 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.710308    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633"} err="failed to get container status \"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633\": rpc error: code = NotFound desc = could not find container \"bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633\": container with ID starting with bfddf266ce384cb975ae726fdadcf1e147568ee6244e07893b563e9568be4633 not found: ID does not exist"
Jul 28 13:17:50.710442 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.710351    2560 scope.go:117] "RemoveContainer" containerID="b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"
Jul 28 13:17:50.710750 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:17:50.710719    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd\": container with ID starting with b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd not found: ID does not exist" containerID="b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"
Jul 28 13:17:50.710852 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.710750    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd"} err="failed to get container status \"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd\": rpc error: code = NotFound desc = could not find container \"b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd\": container with ID starting with b3eac52cd4b201a5b855244c06c4ecd3f4fa4914ee6d428f331d91895439a8bd not found: ID does not exist"
Jul 28 13:17:50.712343 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.712318    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:17:50.718757 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:50.718726    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-13bf1-predictor-5c4bf9fdd5-8977m"]
Jul 28 13:17:52.666351 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:17:52.666317    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" path="/var/lib/kubelet/pods/e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280/volumes"
Jul 28 13:20:10.755750 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:20:10.755621    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:20:10.759642 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:20:10.758558    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:25:10.782793 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:10.782673    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:25:10.784683 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:10.784660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:25:40.386463 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:40.386419    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:25:40.386993 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:40.386746    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" containerID="cri-o://fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922" gracePeriod=30
Jul 28 13:25:40.386993 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:40.386799    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy" containerID="cri-o://c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e" gracePeriod=30
Jul 28 13:25:40.575071 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:40.575038    2560 generic.go:356] "Generic (PLEG): container finished" podID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerID="c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e" exitCode=2
Jul 28 13:25:40.575243 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:40.575101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerDied","Data":"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"}
Jul 28 13:25:42.142009 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.141964    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4fpv/must-gather-t2df6"]
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142448    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142469    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142486    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kube-rbac-proxy"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142493    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kube-rbac-proxy"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142505    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kube-rbac-proxy"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142513    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kube-rbac-proxy"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142533    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142648    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142660    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kserve-container"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142673    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedeb632-087e-4ced-bce7-d8c8a770fecb" containerName="kube-rbac-proxy"
Jul 28 13:25:42.142725 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.142686    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53ddbe5-6d13-4fa0-9eb9-0f5f3847b280" containerName="kube-rbac-proxy"
Jul 28 13:25:42.145957 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.145936    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.148985 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.148963    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4fpv\"/\"kube-root-ca.crt\""
Jul 28 13:25:42.149098 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.148990    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n4fpv\"/\"default-dockercfg-d2g75\""
Jul 28 13:25:42.150192 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.150176    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4fpv\"/\"openshift-service-ca.crt\""
Jul 28 13:25:42.155997 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.155971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4fpv/must-gather-t2df6"]
Jul 28 13:25:42.215289 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.215226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.215289 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.215269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bps\" (UniqueName: \"kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.316040 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.316002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.316238 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.316048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bps\" (UniqueName: \"kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.316376 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.316355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.325204 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.325175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bps\" (UniqueName: \"kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps\") pod \"must-gather-t2df6\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") " pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.468807 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.468767    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:25:42.607926 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.607897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4fpv/must-gather-t2df6"]
Jul 28 13:25:42.610275 ip-10-0-134-201 kubenswrapper[2560]: W0728 13:25:42.610244    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f808972_6b2a_471c_8b55_99c46d963120.slice/crio-1438062f907d9abbef1451e71cb09571ebe40a9764507a4fa6463eca1073ed1b WatchSource:0}: Error finding container 1438062f907d9abbef1451e71cb09571ebe40a9764507a4fa6463eca1073ed1b: Status 404 returned error can't find the container with id 1438062f907d9abbef1451e71cb09571ebe40a9764507a4fa6463eca1073ed1b
Jul 28 13:25:42.612081 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:42.612065    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:25:43.334668 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:43.334554    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 13:25:43.593238 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:43.592425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4fpv/must-gather-t2df6" event={"ID":"9f808972-6b2a-471c-8b55-99c46d963120","Type":"ContainerStarted","Data":"1438062f907d9abbef1451e71cb09571ebe40a9764507a4fa6463eca1073ed1b"}
Jul 28 13:25:48.580709 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.580680    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:25:48.618737 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.618691    2560 generic.go:356] "Generic (PLEG): container finished" podID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerID="fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922" exitCode=0
Jul 28 13:25:48.618951 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.618743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerDied","Data":"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"}
Jul 28 13:25:48.618951 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.618803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" event={"ID":"82a107a6-dd9a-4044-8da6-6b8013f60ad8","Type":"ContainerDied","Data":"64afd7380b0c791cbe0e291cc1b4c13c12ae03f9803272a745f98d803f87c7dd"}
Jul 28 13:25:48.618951 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.618808    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"
Jul 28 13:25:48.618951 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.618833    2560 scope.go:117] "RemoveContainer" containerID="c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"
Jul 28 13:25:48.679839 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.679800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls\") pod \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") "
Jul 28 13:25:48.680066 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.679886    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config\") pod \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") "
Jul 28 13:25:48.680066 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.679916    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kdsk\" (UniqueName: \"kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk\") pod \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\" (UID: \"82a107a6-dd9a-4044-8da6-6b8013f60ad8\") "
Jul 28 13:25:48.680295 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.680263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82a107a6-dd9a-4044-8da6-6b8013f60ad8" (UID: "82a107a6-dd9a-4044-8da6-6b8013f60ad8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:25:48.682438 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.682390    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk" (OuterVolumeSpecName: "kube-api-access-9kdsk") pod "82a107a6-dd9a-4044-8da6-6b8013f60ad8" (UID: "82a107a6-dd9a-4044-8da6-6b8013f60ad8"). InnerVolumeSpecName "kube-api-access-9kdsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:25:48.682588 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.682531    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82a107a6-dd9a-4044-8da6-6b8013f60ad8" (UID: "82a107a6-dd9a-4044-8da6-6b8013f60ad8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:25:48.781908 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.781847    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a107a6-dd9a-4044-8da6-6b8013f60ad8-proxy-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:25:48.781908 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.781896    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:25:48.781908 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.781910    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdsk\" (UniqueName: \"kubernetes.io/projected/82a107a6-dd9a-4044-8da6-6b8013f60ad8-kube-api-access-9kdsk\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:25:48.789816 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.789659    2560 scope.go:117] "RemoveContainer" containerID="fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"
Jul 28 13:25:48.797811 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.797789    2560 scope.go:117] "RemoveContainer" containerID="c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"
Jul 28 13:25:48.798159 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:25:48.798138    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e\": container with ID starting with c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e not found: ID does not exist" containerID="c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"
Jul 28 13:25:48.798217 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.798170    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e"} err="failed to get container status \"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e\": rpc error: code = NotFound desc = could not find container \"c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e\": container with ID starting with c3d96e5345d18e30127e5b70c583b51d8c98a9d6a0dbcf9c1bc9085696d8809e not found: ID does not exist"
Jul 28 13:25:48.798217 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.798191    2560 scope.go:117] "RemoveContainer" containerID="fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"
Jul 28 13:25:48.798424 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:25:48.798411    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922\": container with ID starting with fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922 not found: ID does not exist" containerID="fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"
Jul 28 13:25:48.798469 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.798427    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922"} err="failed to get container status \"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922\": rpc error: code = NotFound desc = could not find container \"fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922\": container with ID starting with fb34b2c2de6c649de204919a1620a0ecd39503e83a39f1fbfbc27728e7b7d922 not found: ID does not exist"
Jul 28 13:25:48.946977 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.946944    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:25:48.951620 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:48.951588    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6"]
Jul 28 13:25:49.335637 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:49.335265    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": context deadline exceeded"
Jul 28 13:25:49.340163 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:49.340123    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dde97-predictor-8465f8b49c-h25g6" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: i/o timeout"
Jul 28 13:25:49.625532 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:49.625434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4fpv/must-gather-t2df6" event={"ID":"9f808972-6b2a-471c-8b55-99c46d963120","Type":"ContainerStarted","Data":"2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf"}
Jul 28 13:25:49.625532 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:49.625484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4fpv/must-gather-t2df6" event={"ID":"9f808972-6b2a-471c-8b55-99c46d963120","Type":"ContainerStarted","Data":"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"}
Jul 28 13:25:49.644371 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:49.644311    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4fpv/must-gather-t2df6" podStartSLOduration=1.395599103 podStartE2EDuration="7.644293595s" totalImagesPullingTime="6.248694492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:25:49.643505692 +0000 UTC m=+3039.478250867" watchObservedRunningTime="2026-07-28 13:25:49.644293595 +0000 UTC m=+3039.479038768"
Jul 28 13:25:50.661686 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:25:50.661592    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" path="/var/lib/kubelet/pods/82a107a6-dd9a-4044-8da6-6b8013f60ad8/volumes"
Jul 28 13:26:10.725315 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:10.725277    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f808972-6b2a-471c-8b55-99c46d963120" containerID="c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b" exitCode=0
Jul 28 13:26:10.725315 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:10.725316    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4fpv/must-gather-t2df6" event={"ID":"9f808972-6b2a-471c-8b55-99c46d963120","Type":"ContainerDied","Data":"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"}
Jul 28 13:26:10.725777 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:10.725579    2560 scope.go:117] "RemoveContainer" containerID="c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"
Jul 28 13:26:11.259153 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:11.259111    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n4fpv_must-gather-t2df6_9f808972-6b2a-471c-8b55-99c46d963120/gather/0.log"
Jul 28 13:26:15.028339 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:15.028303    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-llr2w_785026c7-2c37-44f5-82ae-cc32f614b0d0/global-pull-secret-syncer/0.log"
Jul 28 13:26:15.180320 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:15.180284    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vs7wx_f5bbcca4-dfd3-4609-b22f-88d9d002ae80/konnectivity-agent/0.log"
Jul 28 13:26:15.229570 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:15.229536    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-201.ec2.internal_61ce3bf83fccda127909e8840dbc1840/haproxy/0.log"
Jul 28 13:26:16.738622 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:16.738585    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-n4fpv/must-gather-t2df6"]
Jul 28 13:26:16.739110 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:16.738760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-n4fpv/must-gather-t2df6" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="copy" containerID="cri-o://2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf" gracePeriod=2
Jul 28 13:26:16.741313 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:16.741275    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t2df6\" is forbidden: User \"system:node:ip-10-0-134-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n4fpv\": no relationship found between node 'ip-10-0-134-201.ec2.internal' and this object" podUID="9f808972-6b2a-471c-8b55-99c46d963120" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:16.742357 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:16.742333    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-n4fpv/must-gather-t2df6"]
Jul 28 13:26:16.973357 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:16.973331    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n4fpv_must-gather-t2df6_9f808972-6b2a-471c-8b55-99c46d963120/copy/0.log"
Jul 28 13:26:16.974911 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:16.974890    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:16.977214 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:16.977189    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t2df6\" is forbidden: User \"system:node:ip-10-0-134-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n4fpv\": no relationship found between node 'ip-10-0-134-201.ec2.internal' and this object" podUID="9f808972-6b2a-471c-8b55-99c46d963120" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:17.026503 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.026398    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output\") pod \"9f808972-6b2a-471c-8b55-99c46d963120\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") "
Jul 28 13:26:17.026503 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.026470    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5bps\" (UniqueName: \"kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps\") pod \"9f808972-6b2a-471c-8b55-99c46d963120\" (UID: \"9f808972-6b2a-471c-8b55-99c46d963120\") "
Jul 28 13:26:17.028062 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.028033    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9f808972-6b2a-471c-8b55-99c46d963120" (UID: "9f808972-6b2a-471c-8b55-99c46d963120"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:26:17.028674 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.028645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps" (OuterVolumeSpecName: "kube-api-access-n5bps") pod "9f808972-6b2a-471c-8b55-99c46d963120" (UID: "9f808972-6b2a-471c-8b55-99c46d963120"). InnerVolumeSpecName "kube-api-access-n5bps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:26:17.127996 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.127953    2560 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f808972-6b2a-471c-8b55-99c46d963120-must-gather-output\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:26:17.127996 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.127995    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bps\" (UniqueName: \"kubernetes.io/projected/9f808972-6b2a-471c-8b55-99c46d963120-kube-api-access-n5bps\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\""
Jul 28 13:26:17.758222 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.758194    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n4fpv_must-gather-t2df6_9f808972-6b2a-471c-8b55-99c46d963120/copy/0.log"
Jul 28 13:26:17.758636 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.758543    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f808972-6b2a-471c-8b55-99c46d963120" containerID="2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf" exitCode=143
Jul 28 13:26:17.758636 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.758588    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:17.758709 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.758646    2560 scope.go:117] "RemoveContainer" containerID="2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf"
Jul 28 13:26:17.761087 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:17.761060    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t2df6\" is forbidden: User \"system:node:ip-10-0-134-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n4fpv\": no relationship found between node 'ip-10-0-134-201.ec2.internal' and this object" podUID="9f808972-6b2a-471c-8b55-99c46d963120" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:17.772374 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.772353    2560 scope.go:117] "RemoveContainer" containerID="c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"
Jul 28 13:26:17.773044 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:17.773016    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t2df6\" is forbidden: User \"system:node:ip-10-0-134-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n4fpv\": no relationship found between node 'ip-10-0-134-201.ec2.internal' and this object" podUID="9f808972-6b2a-471c-8b55-99c46d963120" pod="openshift-must-gather-n4fpv/must-gather-t2df6"
Jul 28 13:26:17.783825 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.783804    2560 scope.go:117] "RemoveContainer" containerID="2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf"
Jul 28 13:26:17.784130 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:17.784109    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf\": container with ID starting with 2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf not found: ID does not exist" containerID="2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf"
Jul 28 13:26:17.784206 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.784142    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf"} err="failed to get container status \"2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf\": rpc error: code = NotFound desc = could not find container \"2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf\": container with ID starting with 2dd31928b8048dc45465e0254072b30f6343436925f8ea8455642ba59efb02cf not found: ID does not exist"
Jul 28 13:26:17.784206 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.784166    2560 scope.go:117] "RemoveContainer" containerID="c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"
Jul 28 13:26:17.784406 ip-10-0-134-201 kubenswrapper[2560]: E0728 13:26:17.784385    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b\": container with ID starting with c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b not found: ID does not exist" containerID="c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"
Jul 28 13:26:17.784445 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:17.784413    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b"} err="failed to get container status \"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b\": rpc error: code = NotFound desc = could not find container \"c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b\": container with ID starting with c0e1f2a9c9ec77dc3f623632ed7418ad3d2ead46e7e9a5f27479d4b7d21a785b not found: ID does not exist"
Jul 28 13:26:18.661617 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:18.661582    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f808972-6b2a-471c-8b55-99c46d963120" path="/var/lib/kubelet/pods/9f808972-6b2a-471c-8b55-99c46d963120/volumes"
Jul 28 13:26:19.356455 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.356369    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zm4tt_a383215a-283d-4447-87d4-72ae1f827fb1/kube-state-metrics/0.log"
Jul 28 13:26:19.376732 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.376687    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zm4tt_a383215a-283d-4447-87d4-72ae1f827fb1/kube-rbac-proxy-main/0.log"
Jul 28 13:26:19.404243 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.404214    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-zm4tt_a383215a-283d-4447-87d4-72ae1f827fb1/kube-rbac-proxy-self/0.log"
Jul 28 13:26:19.503101 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.503064    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-52wd5_f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e/node-exporter/0.log"
Jul 28 13:26:19.523086 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.523049    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-52wd5_f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e/kube-rbac-proxy/0.log"
Jul 28 13:26:19.549398 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:19.549375    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-52wd5_f1fbbfb3-1de8-42fe-ba7a-5790c25faf9e/init-textfile/0.log"
Jul 28 13:26:20.060953 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.060913    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-tssq4_5f840ea5-ed93-4a1f-879f-062223498332/prometheus-operator-admission-webhook/0.log"
Jul 28 13:26:20.176312 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.176279    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/thanos-query/0.log"
Jul 28 13:26:20.199703 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.199666    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/kube-rbac-proxy-web/0.log"
Jul 28 13:26:20.222772 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.222743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/kube-rbac-proxy/0.log"
Jul 28 13:26:20.245377 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.245345    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/prom-label-proxy/0.log"
Jul 28 13:26:20.272134 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.272079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/kube-rbac-proxy-rules/0.log"
Jul 28 13:26:20.292668 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:20.292635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff9454757-5ntl5_5ef23623-bf03-4949-a3c7-29a490173793/kube-rbac-proxy-metrics/0.log"
Jul 28 13:26:22.085496 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:22.085463    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/1.log"
Jul 28 13:26:22.095237 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:22.095206    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-66s5m_f811ac55-6afa-4b88-bf84-58957e05b5c2/console-operator/2.log"
Jul 28 13:26:22.571151 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:22.571121    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-857b7978db-lz5xw_509fbcd8-7511-418e-83a0-0f652d196cc1/console/0.log"
Jul 28 13:26:23.111298 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.111267    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-bcjj5_22eb831c-1907-4971-b149-5a1e652035cd/volume-data-source-validator/0.log"
Jul 28 13:26:23.540269 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540232    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"]
Jul 28 13:26:23.540538 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540539    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540549    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="copy"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540554    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="copy"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540563    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="gather"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540568    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="gather"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540588    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container"
Jul 28 13:26:23.540594 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540593    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container"
Jul 28 13:26:23.540845 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540637    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kserve-container"
Jul 28 13:26:23.540845 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540644    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="copy"
Jul 28 13:26:23.540845 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540654    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a107a6-dd9a-4044-8da6-6b8013f60ad8" containerName="kube-rbac-proxy"
Jul 28 13:26:23.540845 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.540659    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f808972-6b2a-471c-8b55-99c46d963120" containerName="gather"
Jul 28 13:26:23.550083 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.550055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.552559 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.552538    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6jls2\"/\"kube-root-ca.crt\""
Jul 28 13:26:23.553743 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.553719    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6jls2\"/\"openshift-service-ca.crt\""
Jul 28 13:26:23.554101 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.553761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6jls2\"/\"default-dockercfg-gtdpd\""
Jul 28 13:26:23.555793 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.555771    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"]
Jul 28 13:26:23.686048 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.686013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-podres\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.686241 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.686071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-lib-modules\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.686241 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.686090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-proc\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.686241 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.686181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtdb\" (UniqueName: \"kubernetes.io/projected/9ecb4fee-df47-410a-a95a-684c951ab838-kube-api-access-xxtdb\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.686241 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.686234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-sys\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787003 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.786964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-podres\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787019    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-lib-modules\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-proc\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtdb\" (UniqueName: \"kubernetes.io/projected/9ecb4fee-df47-410a-a95a-684c951ab838-kube-api-access-xxtdb\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-sys\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-sys\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-podres\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-lib-modules\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.787210 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.787173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ecb4fee-df47-410a-a95a-684c951ab838-proc\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.795604 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.795530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtdb\" (UniqueName: \"kubernetes.io/projected/9ecb4fee-df47-410a-a95a-684c951ab838-kube-api-access-xxtdb\") pod \"perf-node-gather-daemonset-4wbjm\" (UID: \"9ecb4fee-df47-410a-a95a-684c951ab838\") " pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.860666 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.860622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:23.993688 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:23.993660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"]
Jul 28 13:26:23.996102 ip-10-0-134-201 kubenswrapper[2560]: W0728 13:26:23.996071    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9ecb4fee_df47_410a_a95a_684c951ab838.slice/crio-567f5a0ae19db7f12b754c0b4cfc52d6521ace3c65b50e28d35f0ccb7f03b3f9 WatchSource:0}: Error finding container 567f5a0ae19db7f12b754c0b4cfc52d6521ace3c65b50e28d35f0ccb7f03b3f9: Status 404 returned error can't find the container with id 567f5a0ae19db7f12b754c0b4cfc52d6521ace3c65b50e28d35f0ccb7f03b3f9
Jul 28 13:26:24.137672 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.137646    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nkh8w_f933002c-75ff-463e-90c2-e42816673f7f/dns/0.log"
Jul 28 13:26:24.157875 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.157838    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nkh8w_f933002c-75ff-463e-90c2-e42816673f7f/kube-rbac-proxy/0.log"
Jul 28 13:26:24.251492 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.251461    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crz4p_10a4c5eb-9180-464c-85fe-4d7a009b5ab3/dns-node-resolver/0.log"
Jul 28 13:26:24.731835 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.731803    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7bf7d87c9f-99m9w_99dd8cb4-20eb-4851-8301-015e25dac6a6/registry/0.log"
Jul 28 13:26:24.785097 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.785060    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8ncht_3c2cfa72-ee71-41b3-9313-618a23234d37/node-ca/0.log"
Jul 28 13:26:24.788927 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.788895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm" event={"ID":"9ecb4fee-df47-410a-a95a-684c951ab838","Type":"ContainerStarted","Data":"64bd955f80bc9ed6b6b1303a562ebf6213aaa351b7717638b9bc443ee61f321c"}
Jul 28 13:26:24.789146 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.788948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm" event={"ID":"9ecb4fee-df47-410a-a95a-684c951ab838","Type":"ContainerStarted","Data":"567f5a0ae19db7f12b754c0b4cfc52d6521ace3c65b50e28d35f0ccb7f03b3f9"}
Jul 28 13:26:24.789146 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.788970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:24.805765 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:24.805710    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm" podStartSLOduration=1.805692318 podStartE2EDuration="1.805692318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:26:24.804117806 +0000 UTC m=+3074.638863002" watchObservedRunningTime="2026-07-28 13:26:24.805692318 +0000 UTC m=+3074.640437487"
Jul 28 13:26:25.580302 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:25.580259    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7786d8cf-rm5kx_9b0a2c52-afb9-4f06-a3a9-5c5416d72895/router/0.log"
Jul 28 13:26:25.943337 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:25.943306    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jznjs_922664ba-bfbd-4fa5-8d4f-6104e46d0567/serve-healthcheck-canary/0.log"
Jul 28 13:26:26.342784 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:26.342703    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qkxw_36724ce5-e4fa-446d-88de-80d9d42b271d/kube-rbac-proxy/0.log"
Jul 28 13:26:26.364817 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:26.364782    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qkxw_36724ce5-e4fa-446d-88de-80d9d42b271d/exporter/0.log"
Jul 28 13:26:26.389197 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:26.389167    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qkxw_36724ce5-e4fa-446d-88de-80d9d42b271d/extractor/0.log"
Jul 28 13:26:28.752300 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:28.752263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68bc5447d6-wjknz_b4db8684-0aff-4328-a922-9dc0c9e4b218/manager/0.log"
Jul 28 13:26:28.950811 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:28.950769    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-smt2x_d739936f-b0f9-4370-b3ec-cb363b1a3059/manager/0.log"
Jul 28 13:26:28.991884 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:28.991838    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-str2s_55563220-8d75-4524-a5c5-4fe56a39e217/s3-init/0.log"
Jul 28 13:26:29.007777 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:29.007685    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-str2s_55563220-8d75-4524-a5c5-4fe56a39e217/download-hf-model/0.log"
Jul 28 13:26:29.037526 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:29.037495    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-d279j_8a51352a-afb7-41d9-a2c2-5910874654b7/seaweedfs/0.log"
Jul 28 13:26:30.804035 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:30.804001    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6jls2/perf-node-gather-daemonset-4wbjm"
Jul 28 13:26:33.157360 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:33.157324    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm_dcd4d309-870a-4c7c-b841-bcd5bd5235e5/kube-storage-version-migrator-operator/1.log"
Jul 28 13:26:33.159366 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:33.159328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-f2vfm_dcd4d309-870a-4c7c-b841-bcd5bd5235e5/kube-storage-version-migrator-operator/0.log"
Jul 28 13:26:34.206547 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.206516    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7qmmz_8b3873af-4061-4796-ab9a-5c9d44bc5675/kube-multus/0.log"
Jul 28 13:26:34.391004 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.390967    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/kube-multus-additional-cni-plugins/0.log"
Jul 28 13:26:34.411938 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.411906    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/egress-router-binary-copy/0.log"
Jul 28 13:26:34.432951 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.432918    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/cni-plugins/0.log"
Jul 28 13:26:34.453954 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.453927    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/bond-cni-plugin/0.log"
Jul 28 13:26:34.476020 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.475935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/routeoverride-cni/0.log"
Jul 28 13:26:34.496312 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.496287    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/whereabouts-cni-bincopy/0.log"
Jul 28 13:26:34.519313 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.519287    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bvm2g_76047ac2-1c2f-4d1b-be99-9173a606cbef/whereabouts-cni/0.log"
Jul 28 13:26:34.790019 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.789919    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-86zqw_291e2747-1e3c-46df-bd4e-91631ff384f1/network-metrics-daemon/0.log"
Jul 28 13:26:34.806745 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:34.806709    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-86zqw_291e2747-1e3c-46df-bd4e-91631ff384f1/kube-rbac-proxy/0.log"
Jul 28 13:26:35.663937 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.663904    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/ovn-controller/0.log"
Jul 28 13:26:35.696172 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.696140    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/ovn-acl-logging/0.log"
Jul 28 13:26:35.719376 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.719339    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/kube-rbac-proxy-node/0.log"
Jul 28 13:26:35.739677 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.739631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 13:26:35.762904 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.762854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/northd/0.log"
Jul 28 13:26:35.784363 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.784330    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/nbdb/0.log"
Jul 28 13:26:35.804255 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.804223    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/sbdb/0.log"
Jul 28 13:26:35.870341 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:35.870266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9jw6s_3da7d8a9-e9d0-48ab-bccd-bf13478d18f6/ovnkube-controller/0.log"
Jul 28 13:26:37.342189 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:37.342156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ksczq_28edcfdd-cbed-4aad-b0fb-d2d1677c1adc/network-check-target-container/0.log"
Jul 28 13:26:38.289274 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:38.289240    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rpf64_b333236c-930f-4ac0-b950-b02e8e7374a9/iptables-alerter/0.log"
Jul 28 13:26:39.028431 ip-10-0-134-201 kubenswrapper[2560]: I0728 13:26:39.028393    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-sx2xr_80907e86-b3f6-44a8-ae24-dbbb6aae9f57/tuned/0.log"